Million requests per second with Python
medium.com
medium.com
https://en.wikipedia.org/wiki/HTTP_pipelining
> Of all the major browsers, only Opera based on Presto layout engine had a fully working implementation that was enabled by default. In all other browsers HTTP pipelining is disabled or not implemented.[3]
>Internet Explorer 8 does not pipeline requests, due to concerns regarding buggy proxies and head-of-line blocking.[7]
>Internet Explorer 11 does not support pipelining. [8]
>Mozilla browsers (such as Mozilla Firefox, SeaMonkey and Camino) support pipelining; however, it is disabled by default.[9][10] Pipelining is disabled by default to avoid issues with misbehaving servers.[11] When pipelining is enabled, Mozilla browsers use some heuristics, especially to turn pipelining off for older IIS servers.[12]
>Konqueror 2.0 supports pipelining, but it's disabled by default.[citation needed]
>Google Chrome previously supported pipelining, but it has been disabled due to bugs and problems with poorly behaving servers.[13]
That seems like optimizing for a benchmark.
I would imagine "bots" like Googlebot make use of it.
I have been using it daily for 15 years with no problems. I mainly use netcat. This is how I digest large quantities of information as text not possible to do using a browser.
Most websites allow pipelining. Usual max is 100 requests per connection.
But there are rare exceptions where pipelining is disabled. For example, users cannot request pages of results for 100 different queries from Google in a single connection.
Efficient. But not allowed. Always wondered why.
Advertising-filled web pages opened in a "modern browser" are expected to auto-request files from many third party domains. I will call these "conglomerate" pages for lack of a better term.
Not sure HTTP/1.1 pipelining works very well for that. Hence HTTP/2.
HTTP/1.1 benefits users like me. HTTP/2 benefits advertisers, Google, and other companies in the web advertising racket, but not sure how it would benefit users except to serve them advertising and conglomerate pages more efficiently.
As has been posted to HN before the number of outgoing connections that "modern browsers" make to third party servers upon loading a single webpage is staggering. At least, it looks staggering if you have been using the web since the early days, before web advertising grew to its current proportions.
> Efficient. But not allowed. Always wondered why.
The likelihood is that your HTTP requests are getting routed to a backend server/instance/whatever that knows about your query, and can thus return data about that query, but that a different search would get routed somewhere else, to an instance which knows about that query specifically.
> Advertising-filled web pages opened in a "modern browser" are expected to auto-request files from many third party domains. I will call these "conglomerate" pages for lack of a better term.
> Not sure HTTP/1.1 pipelining works very well for that. Hence HTTP/2.
> HTTP/1.1 benefits users like me. HTTP/2 benefits advertisers, Google, and other companies in the web advertising racket, but not sure how it would benefit users except to serve them advertising and conglomerate pages more efficiently.
I think you misunderstand how HTTP/2 works.
HTTP/1.1 pipelining allows you to send five requests to a server, in order, and receive five responses in order. Very straightforward.
HTTP/2 allows you to send any number of requests to a server, in any order, and for the client or server to provide a priority for each of them, and for them to download out of order or in parallel.
What doesn't change between the two is that you're still connecting to one specific server, the same as you were before in HTTP/1.x, and not any number of third party servers, for any reason, in any context. It has zero impact on third-party services like ad networks, trackers, or the like.
client ---(HTTP/2)---> nginx ---(pipelining)---> japronto
Would that not take benefit of japronto's optimisation?
Even if it did, the benefits described in the post (mostly due to making fewer syscalls and having fewer packets to parse) go away because now nginx has to do the work instead of Python. There's probably also some overhead required to collect multiple requests together to pipeline them. You'd just be moving the load around.
And if the nginx process somehow ends up using more CPU than the python process, then using multiple nginx servers would be fairly easy, compared to multiple application servers
Client or even frontend https wouldn't have solved the problem in most cases (it is necessary but not sufficient).
The database (or key value store, etc) replication stream is a prime target, as are any backend protocols that let edge data center storage proxy internal requests for core data centers.
This is very much untrue, especially if you have multiple datacenters, run in the cloud, or operate out of a colo facility.
a) HTTP/2 is often only supported in encrypted mode (h2). E.g. all browsers only supported the encrypted version, and lots of server side libraries then went the same route.
b) encryption might not be needed, if the services already communicate to each other in a secure environment (e.g. are located on the same host, run in an encrypted/secure network, etc.) In such environments the additonal encryption on HTTP level will only show it's downsides, like lower performance and additional deployment/maintainence concerns (certificate deployments, etc.).
I'm no expert in cloud/microservice environments, so I don't know that the usual configuration is there. But in general I would agree that there are some environments where HTTPS is simply not needed, like using it for localhost IPC on trusted systems.
8.1.2.2
Clients MUST also be prepared to resend their requests if
the server closes the connection before sending all of the
corresponding responses.http://www-archive.mozilla.org/projects/netlib/http/pipelini...
8.1.2.2 that you quoted [RFC2616] continues:
> A server MUST send its responses to those requests in the same order that the requests were received.
Perish the thought, we've never seen that before in claims about which languages are fastest
On my machine, using https://github.com/rakyll/hey to test:
1 concurrent request: node.js does ~8000 req/second, go does ~9000, japronto does ~10000
10 concurrent requests: node.js does ~24000 req/second, go does ~45000, japronto does ~55000
Nowhere near a million. That's without pipelining though. Can pipelining really add SO MUCH performance?
UPDATE: tried wrk with pipelining. japronto does 700000, golang 150000, node 35000. Holy shit, pipelining is epic.
But the benchmark is not fair when only one server supports pipelining.
With this mix of C and Python it's impossible to tell what will happen to the system's performance when you actually include significant amounts of business logic written in Python on top of the C framework here. The author even makes a point about how little actual Python code and/or data structures are in use. If use of the host language is discouraged in the framework, how can I trust the performance of the framework with my code written in the host language?
I bet if you put anything non-trivial on top and try to connect with real-world http clients which mostly don't use pipelining, it will fall over hard. Enjoy debugging what you can't understand.
The emphasis on http parsing using SSE intrinsics is odd, as http parsing is rarely a bottleneck. I'm not saying it can't be, but that'll only be when you've got the rest of your stack highly tuned, performance is predictable, and profiling has shown that http parsing is your actual bottleneck. Even so, http/2 in theory alleviates this bottleneck and you don't have to use processor-specific intrinsics, as well as providing a better solution to pipelining problems with its multiplexing of requests.
EDITED: "how can I trust the performance of the framework with my code written in the host language?" Original text lacked emphasized addition.
The point was that Go's http stack and standard library and runtime being written all in Go gives me confidence that any Go code I write on top will enjoy the same performance characteristics and that there will be very few surprises. It's predictable due to its uniformity.
So CPYTHON can not in fact deliver those performances. But CPYTHON can be extended through Japronto to achieve them. I'd say it's still misleading, since casually when we refer to Python, it means CPYTHON.
Go's primary implementation provides all the fast goodies needed for fast performance. There could be a slow Go compiler which didn't, but when you say Go, it refers to Google's implementation.
You could argue that at least, CPYTHON makes it easy to extend the language with C. So Python language extensions are trivial to use compared to some other languages. So at least I'll give you that.
But anyway… HTTP/2, or 1.1 pipelining for that matter, is usually terminated at the reverse proxy level. So it's really not necessary in a web framework! Just makes these unfair benchmark results possible.
If you look at the Plaintext TechEmpower benchmark [1], for instance, the "echo-prefork" Go benchmark hits 3.6M requests per second. fasthttp hits 2.8M. On a less powerful cloud server, fasthttp hits 850000 and echo-prefork hits 742000 (yes, fasthttp is faster on a slower system...the fun of benchmarks).
Not sure how fast your machine is, but I'm sticking with Go for my performance-critical code. As a parallel comment points out, Go is also fast no matter what code you're running, and Go's optimizations are throughout the stack, including some pretty extreme garbage collection optimizations, so when you have a complex, long-running server, GC won't be killing latency at a crucial time.
[1] https://www.techempower.com/benchmarks/#section=data-r13&hw=...
In that case, the author is measuring something that's completely useless. The entire advantage of Go is that it has really good multithreaded asynchronous behavior.
A benchmark that's measuring single thread performance of a task that's optimized to do well in a single thread (i.e., it doesn't actually do anything other than return static text) is entirely worthless.
If you're going to be returning static text, may as well compare to Nginx, which I'm certain can return static text even faster. If you're going to be doing processing in Python, then do at least something.
And run in multiple threads, since that's Go's native environment.
Rating: PANTS ON FIRE. [1]
[1] Not your comparison, but the original author's performance claims.
Anyone know why go went from 45kreq/s to 150 when pipelining was enabled - without supporting pipelining? Or is your last run with yet more concurrent requests?
1) client: gimme 10
2) server: here, have 1
3) client: ok, gimme 9!
4) server: here, have 1
...
?It's more like this:
1) client: gimme 10, gimme 9...
2) server: here, have 10
3) server: here, have 9
...
Who cares? The author wrote the C code and Python bindings for his library so we can _all_ benefit from handling more requests per second just by writing Python code and using his framework.
I appreciate the work and will be exploring its potential use. Thank you squeaky_pl for writing it.
You could give the same treatment to virtually any other host scripting language (Ruby, Lua, PHP) and then make outlandish claims that that host language is now capable of ridiculous things when it had nothing to do with the alleged success other than that it exposes a FFI that lets you create something like this.
"All the techniques that were mentioned here are not really specific to Python. They could be probably employed in other languages like Ruby, JavaScript or PHP even. I would be interested in doing such work as well but sadly this will not happen unless somebody funds it."
Making extensions in C and FFI are a feature of Python. Should we then say that the scientific community is not really using Python because it's using all the C modules in numpy and other optimization library to do all that fast work?
> You could give the same treatment to virtually any other host scripting language (Ruby, Lua, PHP) and then make outlandish claims that that host language is now capable of ridiculous things when
Nothing wrong with it. Then you'd have a fast framework in that language two. You can have 3 fast frameworks in 3 different scripting languages. That is totally ok.
This ruby benchmark doesn't matter because ruby is written in C!
This nodejs benchmark doesn't matter because v8 is written in C/C++!
etc.
No it can't. Each of those languages you listed (as well as Python) has its own runtime system designed to implement the desired semantics and features of the language. When writing a C extension to one of these runtimes, you're escaping the confines of the runtime system in order to improve performance or capabilities not native to the host runtime.
The point isn't "because it's written in C"; the point is that it's not written in Python and therefore the results of the benchmark cannot be attributed to Python, to say nothing of the merit of the benchmark in the first place.
I was able to do network streaming (including live encoding) of audio and video in Python. Do you think that would be possible if it was native python code? Of course not, I used GStreamer which did that in C. My objective was not to write an encoder but stream the video, so as far as I am concerned my code was in Python.
One of big strengths of Python is that when you need to improve its performance you just need to locate bottlenecks and rewrite them in a faster language. This is especially great when you want to do it in code that is not your primary objective, like in this project, you as a developer care what your code returns on specific requests you don't really care about how service handles each request etc. This is what a high level programing language is about.
Except it's all exposed via a python API that can be easily installed with pip and you can use it without even realizing that you're calling C.
This is how python was always supposed to be optimized.
$ time python -c "import json;
for i in range(0, 1000000):
json.loads('{\"data\": \"hello, world\"}')"
real 0m7.055s
user 0m6.951s
sys 0m0.063s
As it stands, the title of the post would be more accurately stated as 'million requests per second with C'.
$ time python -c "import json;
for i in range(0, 1000000):
json.loads('{\"data\": \"hello, world\"}')"
real 0m3.142s
user 0m3.098s
sys 0m0.034s
$ time python -c "import ujson as json;
for i in range(0, 1000000):
json.loads('{\"data\": \"hello, world\"}')"
real 0m0.444s
user 0m0.399s
sys 0m0.028s
Also, just because I was curious:
$ time pypy -c "import json;
for i in range(0, 1000000):
json.loads('{\"data\": \"hello, world\"}')"
real 0m0.422s
user 0m0.386s
sys 0m0.032sOf course, most websites aren't dealing with anywhere near that volume of requests, and then Python is fine.
Pypy is interesting, but last time I looked it didn't support C extensions and hence can't be used with the poster's library.
$ python -m timeit --setup 'import json' "json.loads('{\"data\": \"hello, world\"}')"
100000 loops, best of 3: 9.34 usec per loop0.291s
Well, that was an easy way to reduce 7 seconds to 0.3.
Java Golang Rust Scala Maybe C# C
The semi famous techempower benchmarks shows several popular frameworks that can handle over 1mill/sec
My statement was that it could handle the load. Taken into account the fact that the language is built around concurrency, it would handle the load within multiple threads without a hitch.
Say what? When I do this kind of testing using wrk:
1. I use multiple clients
2. Each client uses a thread for every logical CPU
3. I usually use at least 256 connections
No to mention in regards to Go/Others:
1. He is using an RC
2. By default it uses all all cores, why handicap it?
3. He didn't share his code for his contestants, how can I trust this?Which is silly considering a majority of wild browsers now support HTTP/2.
Thanks for pointing our the repo, I missed it before: https://github.com/squeaky-pl/japronto
His Go isn't horrible, but it could be written like this:
func hello(respWtr http.ResponseWriter, req *http.Request) {
text, status := []byte("Hello world!"), http.StatusOK
if request.URL.Path != "/" {
text, status = []byte("Not Found"), http.StatusNotFound
}
respWtr.WriteHeader(status)
response.Write(text)
}
Also, limiting Go to use a single core is unfair because the Go GC is optimized for multi-core operation.
"optimised for loopback!" - which is nobody's actual problem.
It's a cool thing though, just cynical I guess because it doesn't solve my actual problems which are all self inflicted and organisational. Thinking about deleting this comment because it would logically apply to every incremental technical advance :(
The author's claim that the other http servers are deficient is misleading.
Along those lines, what happened to uwsgi async mode? uwsgi has been rock solid for me, with a straightforward binary framing and hot loading, and it would be great to use it for the (admittedly, few) cases where async might be useful. It seems like one of those obvious good ideas that no one uses?
Much more thorough than this toy benchmark article, which reads like marketing release...
If you want to make your chosen platform fast you can see what they're doing in the src to get those results. The ones not marked "stripped" and with "full" for ORM are very realistic examples. Not sure where you're going with this? Nobody's real workload looks exactly like benchmarks.
1 Go worker process can utilise all the cores in the server. I just want to get that clarified.
> To be fair all the contestants (including Go) were running single worker process
That seems to imply that they were all limited to a single core.
This makes the benchmark results much less impressive. Sure its cool that this micro framework allows you to serve a million requests per second with a single process but no one is going to be running a single process per server in production if they value their uptime.
I don't suppose the hand tuned microhttp parser supports h2?
Pipelining is a dead technology that was never really alive. For those who aren't familiar with it, it's based on a http "hack" where after http/1.1 was released and response end could be signaled via something other than EOF on the tcp connection they figure you can send n requests in a row and receive n responses in a row. This was in the days where good manners required you make no more than 2 simultaneous requests to a server.
Nobody ever really supported it, and the lack of support for out of order responses meant that any one slow request could slow down everything else. The h2 model allows at least hundreds of simultaneous requests and responses multiplexed over one TCP stream.
I would pay for that.. a lot (as much as I can afford). And I'm pretty sure so would a lot of others.
What kind of data do people look at when they want to prove their language is better, or choose the best language?
To make the benchmark a bit fair, add a graphic of memory usage, pipelining might have a huge impact on resources usage in the server side.
Use as many appservers as your customer base will pay you for.
Basically, cry in your beer because you're so rich and successful now that the framework won't handle your stuff.
Spoiler: technically, it's not "with Python".
In any case, this is written in C so I'm not sure it proves anything about the Python language.
You can scale up whatever code you want in whatever language, you just need to structure it in a way that allows for that.
> It lets you do synchronous and asynchronous programming with asyncio and it’s shamelessly fast. Even faster than NodeJS and Go.
and then,
> To be fair all the contestants (including Go) were running single worker process.
What is the point of such a benchmark? It is completely useless. You are deliberately crippling other tech before comparing your tech with them.
What am I missing here HN?
allocating significantly different compute resources for different samples would be at least as laughable (eg. giving Go a dozen cores [per it's defaults] but constraining Python to one worker [per it's defaults]).
Sounds like you have a long and fun road ahead of you with this project.
If that is correctly understood then the benchmark is highly artificial.
But yeah, feel free to correct me if I got it wrong.
I'm getting slammed for my previous comment!
But then running the demo app resulted in:
ImportError: dlopen(/tmp/japronto/src/japronto/protocol/cprotocol.cpython-35m-darwin.so, 2): Library not loaded: libpicohttpparser.so
Referenced from: /tmp/japronto/src/japronto/protocol/cprotocol.cpython-35m-darwin.so
Reason: image not found
Actually, I have a friend of mine running a streaming website. He banks 15k€ a month for 400k daily users.
Python perf has zero inpact on his project. Python ease of use and maintainability means he, as a terrible (really terrible) programmer have been able to keep the site up and running for 5 years.
I worked in africa for a while. Needs for Python perf ? Zero. Needs for code that is understandable and easy to write. A LOT.
Moved to geographes working on SIG. Needs for Python perf ? Hell the GIS engine does the heavy work, they don't care. But they don't undestand jack about programming and need things done. Python let them to do it cleanly.
Or I moved to embeded programmers. Hardcore C coders that are using Python to test their boards. Need for Python perf ? They don't even know what you are talking about. For them perf is assembly. They want something that is fast to write, with 10000 libs available.
As a trainer and dev I teach Python on a monthly basis. JS training. Jeez, where do I begin ? If you let them do something more than a one-file stuff, they are lost. Python ? They are project ready in no time.
Data mongers ? They have numpy and databases.
Now myself ? I code in Python all the time. I've seen many slow sites. It's always either because of the DB queries or the static assets. Never about the server side language, be it Python, JS, Ruby or PHP.
Coding in Go, Elixir or Rust for task where my productivity is more important than the machine output makes no sense. Especially in a world where an unmetered VPS cost 3€/month (https://www.ovh.com/us/vps/) with 2 GB RAM and 10 GB SSD. I can just take 100 of those and build a freaking skynet without making my company accountant blink.
We are not google or facebook (and they do use Python BTW. Fb just announced they are migrating to 3 massively).
So yes, more perf is nice. I would literally pay to help having better perf because it's always a good thing. But stop pretending you need it so bad, because you would be magically the 0.1% that actually do on all the forum post where people are saying they are.
It's a nice to have.
If you want to switch to something else, just admit you are following a trend. Dev always are. They love what's new and shiny. Nothing wrong about it. But those "Python is slow I'm out" posts are ridiculous. Dropbox can say that. Bank of America can say that.
It would be interesting to see an analysis done of developer productivity across languages to find out which language provides the best over all value for new and existing codebases.
But Python provided me with a better experience than ruby because:
- it's very useful outside of the Web as well. Ruby is still very much tied to RoR.
- the Python community value documentation, tests and dislike too much magic, like monkey patching.
- as a trainer, making people setup Python + venv and make them use it is wayyyy easier than rvm.
Then second to them would be JS. It's a terrible language, but it's the only one on the browser platform which is amazing. Having code that natively manipulate GUI, sound and video in a few line is the most motivating experience you can get when you start coding.
Create an array or string, take the last element and display it:
<A B C>[* - 1].say;
The same in Python:
print(['A', 'B', 'C'][-1])
The same in Ruby:
print ['A', 'B', 'C'][-1]
The first version is a good example of what perl is good at : it's short to write. The Python version is more verbose.
But, your brain has to do the following:
- Scan the element to realise A, B and B are strings. Perl can have strings with "A", but in this here you can write it without it. So when you read, you need context to know what's going on. The Python version make it clear it's a string.
- Then figure out the that the separation is on the spaces. Python put comas, so you don't have to parse negative space in your head.
- the you have the "*" you need to integrate to get context of what element you use. Then you get the ";" which is just noise.
Small elements like those add up very, very quickly in a code base. Every time your brain has to gather context before understanding what it reads, you add cognitive load. You can't scan the code.
PHP, Perl and JS by nature have a higher cost that Ruby or Python. The laters have been design with readability in mind. You have less tricks, they are more boring. And easier to read.
Of course you have many other small issues with perl, like figuring out why you use .say but .WHAT in uppercase. Why the shell by default on some linux setup have a broken history. Why you can leave out the .say most of the case in the shell, but sometime if you don't use it nothing is displayed. That you have to install the rakuto package on Ubuntu but run the perl6 command to start the shell, etc.
All in all, Perl is a neat technical achievement, but has not been very ergonomy oriented.
C type brackets and semicolons are second nature to a lot of people.
Perl has a lot of built-in stuff that makes a lot of sense when you have learned it.
Your example in Perl 5:
print qw/A B C/[-1];
Can be written like this if you like the commas and quotation marks:
print qw/'A', 'B', 'C'/[-1]; #Will warn
print(('A', 'B', 'C')[-1]);
say qw/A B C/[-1];
print qw/A B C D/[0..2];
my ($a, $b, $c) = qw/A B C/;
print my $num =()= qw/A B C/;
Take a look at some code in Python that is done by someone less disciplined (or intentionally obfuscated). It is as unreadable (or worse) as the obfuscated C and Perl from the late 90s.
Using so much context to convey meaning implying a lot of processing even if you are train for it, so you have an additional load. Using so many symbols to convey meaning as well.
By design, Perl is harder to read if you write it the way it's intended to be used.
Just like Russian is harder to read than english because the language has some concept like letter effecting other letters than implies you brain need to backtrack.
Eventually, if you become a master at it, you won't feel it. But you can only master so many things in life. Most of the stuff, at best, we become very good at it. But very good means you still use your brain CPU. There is no way around it.
That's why language design is important, and to me, Perl failed this big time.
It's not just about habit either. Take Erlang for example. It is a very different language. It's hard to reason about. But it's not very hard to parse in your head because it's quite consistent and regular in the way it unfolds.
Erlang has other issues of course and I wouldn't recommand it over Perl for non highly // related tasks.
But yes, Perl has been written to be clever. This is not a quality in language design. Handy yes. Intelligent yes. Clever, not so much.
$ - a single thing
@ - a list of things
% - a hash (associative array, key-value pair list)
\ - a reference to any of the above
my @fruits = qw/Apple Pear/;
my $company = $fruits[0];
# A reference can be declared directly, this is a \%
# with a nested \@. Encodes nicely to JSON as an example.
my $hash_ref = {
fruits => [
'Apple',
'Pear'
]
};
my $apple = $hash_ref->{fruits}[0];
What way is it intended to be used? Perl 4 style mixing Perl and shell script?
Perl 5 has had (roughly) yearly releases since 1994 and is now at 5.24. Quite a bit has changed in that time and the language has evolved and improved.
There are modern non blocking web frameworks, great database support, several modern OO systems etc.
And Perl still works great shell script style (you can choose to code in a clean style) and for the most parts there is great backwards compatibility.
Perl is definitively about getting the job done. Hence the slogans "make easy things easy and hard things possible" and "there is more than one way to do it".
For someone well versed in Perl something like idiomatic Python becomes a prison where it is hard to get the job done because you can't freely express yourself.
I think it's not just philosophy or language design but people are wired differently. And Perl is a lot of fun for creative individuals.
Lastly, I am looking forward to the next years when the Python community has to clean up because of its current popularity with novice programmers. That's what Perl has been through and that old novice code is also where most of the undeserved reputation come from.
Highest level language that's maintainable wins. That's why Python is incredibly popular. The real snag to Python is the flippant disregard to backwards compatibility even though it's already massively popular. Fragmenting the ecosystem.
Wow. Which streaming website is this?
For the by far largest share of web applications, easily everything to the 99th percentile, performance, in the "sense" of this benchmark, is wholly irrelevant.
Amen to that. Performance is incredibly low on the list of concerns for most businesses I've worked for. Getting code out quickly and making it work reliably are usually priorities #1 and #2.
Performance is a "sexy" problem though.