Benchmark issue

[felipetrz]

maybe someone with rust experience can have a look at my wrk/rust/hello code and tell me why it's surprisingly slow, as I expected it to be faster than the basic GO example.

Since you asked, both your Zap and Go samples are doing evented async IO (facil.io is doing the heavy lifting for you, and Go's support for concurrent async IO is it's main feature).

Both your Python and Rust comparisons are unfair, as they're just using blocking threaded IO.

For a more fair comparison, I suggest you replace your Python sample (which is likely using Flask) by the equivalent Starlette implementation, and your inefficient Rust sample by an Axum application.

[renerocksai]

Thanks! I'll definitely check that out and add those results to the README for clarification! I'm fairly confident I'll get the python code to work but I am a total rust noob. Will report back if I need more help.

[renerocksai]

Ah, there seem to be easy examples for both of them. Will check them out.

[renerocksai]

BTW the python example is using the built-in http.server module. Flask apps I've usually deployed with gunicorn but the example should be just a simple Request -> Response so I thought introducing Flask would be an overkill for this specific test.

[felipetrz]

In this case, a simpler async version, without a lot of framework overhead, would be using aiohttp

There are quite a few faster options as well: http://klen.github.io/py-frameworks-bench/

In fact, blacksheep seems to hit numbers that are pretty close to your go and zap results, so that could be an interesting comparison under the same hardware conditions.

[renerocksai]

Thank you for pointing out a few more options for python. As I wrote here, I initially wanted it to be an equally 'unfair' comparison, not a comparison of the most optimized versions out there. Blacksheep and Starlette, for example, I will put on the back burner because they require a separate ASGI HTTP server if I understood correctly. So what would I test against then?

I wanted to compare: what performance you get when you know nothing and just use the default way of going about things. Of course, this line of thinking is flawed, but it's the one I used. In go, I compare against a basic GO implementation. In python, I compare against a basic python implementation. In rust, I compared against a basic rust implementation.

However, I am still interested in comparing to more sophisticated alternatives. But I also want to avoid just comparing C wrappers, at least initially. Checking out the benchmark site you provided, I am going to try sanic since it doesn't require a separate server process like gunicorn. It's also supposed to be way faster than aiohttp, so I think that's fair.

[renerocksai]

Hmmm, just realized sanic comes with its own launcher process, too. I don't like that. It doesn't feel like starting a python script. [Edit]: It still allows me to start it with the python interpreter, launching whatever processes it needs, so I'll keep it 😄

[renerocksai]

Thank you, @felipetrz ! I have added the new timings to the README. Have a look! The rust side of things looks much better, more realistic now. All the source code is there for you to check, as well. It's in the ./wrk subdirectory.

[renerocksai]

Since this topic comes up again and again, I link issue #12 here for more.

To me, with axum/tokio (introduced in d7946ec), the "rust looking bad" problem is addressed enough. It's a fair comparison now in terms of comparing optimized, current frameworks.