Async Python is not faster

Sure, the operating system can find other things to do with the CPU cycles when a program is IO-locked, but that doesn't help the program that you're in the situation of currently trying to run.

> An async implementation that does multiple ("embarrassingly parallel") tasks in the same process - whether that is DB IO waiting or microservice IO waiting - is not necessarily a performance improvement over a sync version that just starts more workers and has the OS kernel scheduler organise things. In fact in practice an async version is normally lower throughput, higher latency and more fragile. This is really what I'm getting at when I say async is not faster.

You're right. "Arbitrary programs will run faster" is not the promise of Python async.

Python async does help a program work faster in the situation that phodge just described (waiting for web requests, or waiting for a slow hardware device), since the program can do other things while waiting for the locked IO (unlike a Python program that does not use async and could only proceed linearly through its instructions). That's the problem that Python asyncio purports to solve. It is still subject to the Global Interpreter Lock, meaning it's still bound to one thread. (Python's multiprocessing library is needed to overcome the GIL and break out a program into multiple threads, at the cost that cross-thread communication now becomes expensive).

On the web when the bulk of your application code time is waiting on APIs, database queries, external caches or disk I/O it creates a dramatic increase in the capacity of your server if you can do it with minimal RAM overhead.

It's one of the big reasons I've always wanted to see Techempower create a test version that continues to increase concurrency beyond 512 (as high as maybe 10k). I think it would be interesting.

This is very true, especially when actual work is involved.

Remember, the kernel uses the exact same mechanism to have a process wait on a synchronous read/write, as it does for a processes issuing epoll_wait. Furthermore, isolating tasks into their own processes (or, sigh, threads), allows the kernel scheduler to make much better decisions, such as scheduling fairness and QoS to keep the system responsive under load surges.

Now, async might be more efficient if you serve extreme numbers of concurrent requests from a single thread if your request processing is so simple that the scheduling cost becomes a significant portion of the processing time.

... but if your request processing happens in Python, that's not the case. Your own scheduler implementation (your event loop) will likely also end up eating some resources (remember, you're not bypassing anything, just duplicating functionality), and is very unlikely to be as smart or as fair as that of the kernel. It's probably also entirely unable to do parallel processing.

And this is all before we get into the details of how you easily end up fighting against the scheduler...

Co-routines are not necessarily faster than threads, but they yield to a performance improvement if one has to spin thousands of them : they have less creation overhead and consume less RAM.

Literally the first thing any concurrency course starts with in the very first lesson is that scheduling and context overhead are not negligible. Is it so hard to expect our professionals to know basic principles of what they are dealing with?

This is because when they are first shown it, the examples are faster, effectively at least, because the get given jobs done in less wallclock time due to reduced blocking.

They learn that but often don't get told (or work out themselves) that in many cases the difference is so small as to be unmeasurable or in other circumstances the can be negative effects (overheads others have already mentioned in the framework, more things waiting on RAM with a part processed working day which could lead to thrashing in a low memory situation, greater concurrent load on other services such as a database and the IO system it depends upon, etc).

As a slightly of-the-topic-of-async example, back when multi-core processing was first becoming cheap enough that it was not just affordable at give but the default option, I had great trouble trying to explain to a colleague why two IO intensive database processes he was running were so much slower than when I'd shown him the same process (I'd run them sequentially). He was absolutely fixated on the idea that his four cores should make concurrency the faster option, I couldn't get through that in this case the flapping heads on the drives of the time were the bottleneck and the CPU would be practically idle no matter how many cores it had while the bottleneck was elsewhere.

Some people learn the simple message (async can handle some loads much more efficiently) as an absolute (async is more efficient) and don't consider at all that the situation may be far more nuanced.

You mean concurrent tasks in the same process?

I do.

And I don't think I'm alone nor being unreasonable.

I think rather the point is to make your APPLICATION either finish in less time, or to not take MORE time when given more load.

The code runs as fast as it runs, coroutines notwithstanding.

The author of "black" suggested that the cause of the slow down may be that asyncio actually starved postgres for resources:

https://twitter.com/llanga/status/1271719783080366086

That is not true, at least not in general, the whole point of using continuations for async I/O is to avoid the overhead of using threads, the scheduler overhead, the cost of saving and restoring the processor state when switching tasks, the per thread stack space, and so on.

The use of async either as callbacks, or user threads, or coroutines, is a convenience layer for structuring your code. As I understand, that layer does add some overhead, because it captures an environment, and has to later restore it.

I was thinking this would be about using multiprocessing to fire off two or more background tasks, then handle the results together once they all completed. If the background tasks had a large enough duration, then yeah, doing them in parallel would overcome the overhead of creating the processes and the overall time would be reduced (it would be "faster"). I thought this post would be a "measure everything!" one, after they realized for their workload they didn't overcome that overhead and async wasn't faster.

Upon what the post was about, my response was more like "...duh".

You are making dinner. You start to boil water for the potatoes. While that happens, you prepare the beef. Async.

You and your girlfriend are making dinner. You do the potatoes, she does the beef. Parallel.

You can perhaps see how you could have asynchronous and parallel execution at the same time.

In the context of a Web server, a request is handled by a single Python process (so don’t give me that “OS scheduler can do other things”). Async matters here because your request turnover can be higher, even if the requests/sec remains the same.

In the cooking example, each request gets a single cook. If that cook is able to do things asynchronously, he will finish a single meal faster.

If it were only parallel, you could have more cooks - because they would be less demanding - but they would each be slower.

> “Faster” is misleading

and

> "At low levels there is going to typically be negligible or no speed gains, but at higher levels the benefit will be incredibly obvious."

there are no "speed" gains period. the same amount of work will be accomplished in the same amount of time with threads or async. async makes it more memory efficient to have a huge number of clients waiting concurrently for results on slow services, but all of those clients walking off with their data will not be reached "faster" than with threads.

the reason that asyncio advocates say that asyncio is "faster" is based on the notion that the OS thread scheduler is slow, and that async context switches are some combination of less frequent and more efficient such that async is faster. This may be the case for other languages but for Python's async implementations it is not the case, and benchmarks continue to show this.

The point is that async IO allows your own process/thread to progress while waiting for IO. Preemptive multitasking just assigns the CPU to something else while waiting, which is good for the box as a whole, but not necessarily productive for that one process (unless it is multithreaded).

In doing that, you're removing natural parallelism, and end up competing with the kernel scheduler, both in performance and in scheduling decisions.

Node's JIT comes from a web browser's javascript implementation used by billions of people. It's also had async baked in from day one.

Python started single process, added threading, and then bolted async on top of that. And CPython is a pretty straight interpreter.

A comparison between Node and PyPy would be more informative, but PyPy has a far less mature JIT and still has to deal with Python's dynamism.

> If we invent a label maybe they’ll start becoming self aware and start acting more down to earth.

You can't lecture people into self-awareness, any more than experts can lecture everyone into wearing masks.

Doesn't this prove that async is waiting for connections when you put a limit on it? The only way async wins is if it is free to hit the db whenever it needs to.

until then I'd had pretty much bought the hype that the new async frameworks running on Uvicorn were the way to go

I'm very glad to see this kind of comparative test being made, it's very useful, even if it later gets refined and added to and the results more nuanced

Highly disagree as the database is just another IO connection to a server, which is asyncio bread and butter. Being able to stream data from longer running queries without buffering and whilst serving other requests (and making other queries) is really quite powerful.

But yeah, if you're maxing out your database with sync code then async isn't going to make it magically go faster.

this is exactly why the issue is so concerning for me as well.

Ok in 2015 it was a pain but with Python 3.8 it's actually a only joy & fun in my opinion.

> the balkanisation of the community could well be more problematic than 2 vs 3

If you could call Python2 code from Python3 or vice-versa as easily as you can do with async then it would be comparable.

Yeah, that's an extremely painful way to write threaded code. Much more normal is to simply block your thread while waiting for others to .Join() and return their results, likely behind an abstraction layer like a Future.

The only time you really need to use callbacks is when you need to blend async and threaded code, and you aren't able to block your current thread (e.g. Android main thread + any thread use is an example of this). But there are much much easier ways to deal with that if you need to do it a lot - put your primary logic in a different, blockable thread.

That's not how it works. `async` and `await` are merely syntactic sugar around callbacks. Everything in javascript is already nonblocking[1], whether or not you use async/await.

[1] There are a few rare exceptions in node js (functions suffixed with "Sync"), but in the same vein, they are blocking whether or not you use async/await.

Taken from Stephen Cleary's SO answer on this topic: https://stackoverflow.com/a/31192718

Memory is cheap; the cost is in constant de/serialization. Same with "just rewrite the hotspots in C!"-style advice; de/serialization can easily eat anything you saved by multiprocessing/rewriting. Python is a deceivingly hard language, and a lot of this is a direct result of the "all of CPython is the public C-extension interface!" design decision (significant limitations on optimizations => heavy dependency on C-extensions for anything remotely performance sensitive => package management has to deal extensively with the nightmare that is C packaging => no meaningful cross-platform artifacts or cross compilation => etc).

Maybe what you're getting at is cases where there are a large number of (fairly sleepy) open connections? Eg for push updates and other websockety things. I didn't test that I'm afraid. The state of the art there seems to be using async and I think that's a broadly appropriate usage though that is generally not very performance sensitive code except that you try to do as little as possible in your connection manager code.

Concurrency can help you separate out logic that is often commingled in non-concurrent code, but doesn't need to be. As a real-world example, I used to do safety critical systems for aircraft. The linear, non-concurrent version, included a main loop that basically executed a couple dozen functions. Each function may or may not have dependencies on the other functions, so information was passed between them over multiple passes through this main loop (as their order was fixed) using shared memory.

A similar project had about a dozen processes, each running concurrently. There was no speed improvement, but the connection between each activity was handled via channels (equivalent in theory to Go's channels, less like Erlang's mailboxes as the channels could be shared). We knew it was correct because each process was a simple state machine, separated cleanly from all other state machines.

The second system's code was much simpler, there was no juggling (in our code) of the state of the system, compared to managing the non-concurrent logic. If a channel had data to be acted on, the process continued, otherwise it waited. Very simple. And it turns out that many systems can be modeled in a similar fashion (IME). Of course, we had a very straightforward communication mechanism (again, essentially the same as Go channels except it was a library written in, as I recall, Ada by whoever made the host OS).

Given how slow I/O operations are, and how much modern code depends on the network, we typically need some concurrency in our code. So for me, almost always, the question isn't, "which concurrency choice is fastest?" but rather, "which concurrency choice is fast enough while leading to code with the least bugs?"

Yes and no. Again, I have not profiled or optimized servers or interpreted/JIT languages, so I bet there's a new ruleset.

Blocking can come from unexpected places. For example, if we use dependencies, then we don't have much control over the resources accessed by the dependency.

Sometimes, these dependencies are the OS or standard library. We would sometimes have to choose alternate system calls, as the ones we initially chose caused issues which were not exposed until the profile was run.

In my experience, the killer for us was often cache-breaking. Things like the length of the data in a variable could determine whether or not it was bounced from a register or low-level cache, and the impact could be astounding. This could lead to remedies like applying a visitor to break up a [supposedly] inconsequential temp buffer into cache-friendly bites.

Also, we sometimes had to recombine work that we had sent to threads, because that caused cache hits.

Unit testing could be useless. For example, the test images that we often used were the classic "Photo Test Diorama" variety, with a bunch of stuff crammed onto a well-lit table, with a few targets.

Then, we would run an image from a pro shooter, with a Western prairie skyline, and the lengths of some of the convolution target blocks would be different. This could sometimes cause a cache-hit, with a demotion of a buffer. This taught us to use a large pool of test images, which was sometimes quite difficult. In some cases, we actually had to use synthesized images.

Since we were working on image processing software, we were already doing this in other work, but we learned to do it in the optimization work, too.

When my team was working on C++ optimization, we had a team from Intel come in and profile our apps.

It was pretty humbling.

We've had a track record of technologies which:

1) Automated things (reliving programmers from thinking about stuff)

2) Were expected to make stuff slower

3) In reality, sped stuff up, at least in the typical case, once algorithms got smart

That's true for interpreted/dynamic languages, automated memory management/garbage collection, managed runtimes of different sorts, high-level descriptive languages like SQL, etc.

Sometimes, it took a lot of time to figure out how to do this. Interpreters started out an order-of-magnitude or more slower than compilers. It took until we had bytecode+JIT that performance roughly lined up. Then, we started doing profiling / optimization based on data about what the program was actually doing, and potentially aligning compilation to the individual users' hardware, things suddenly got a smidgeon faster than static compilers.

There is something really odd to me about the whole async thing with Python. Writing async code in Python is super-manual, and I'm constantly making decisions which ought to be abstracted away for me, and where changing the decisions later is super-expensive. I'd like to write.

    result = asyncio.run(red(x))

Not seeing any particular readability issue with that usage neither. If you don't call asyncio.run or await on the result of an async function call, then you get a coroutine for result.

For instance, imagine a "maybe_await" method that just calls sync if is synchronous or otherwise awaits.

For example, when someone access a descriptor in Django.. this could end being a query to the db (transparent) but dangerous. With asyncio you explicitly await something to return the execution to the event loop.

At least for me sounds like a safer behaviour

Oh interesting, are you saying an intelligent forking implementation is able to share static portions of memory with multiple children?

I was perhaps under the naive assumption forking was pretty much just a full memory copy of the parent.

uvicorn --port 8001 --workers $PWPWORKERS app_starlette:app --loop uvloop

The uvicorn docs should point out what a big difference uvloop makes.

If I'm not wrong, aiopg it's something not fully async, just because relais on the old driver.

Edit: I've been downvoted so I'll add a precision: Usually, it is believed that async shines against other models once you reach a certain scale (https://en.wikipedia.org/wiki/C10k_problem). This benchmark shows than async app frameworks are slower than the sync ones when running at a given scale, and since the article doesn't give many details on the incomming traffic, I can only assume that it's low, since it saturates 4 cores.

I believe that your conclusion that "Async python is not faster" is an over generalization of your use case.

I'm not saying that the configuration in your benchmark is not correct, I am saying that this benchmark may not yield the same results if you try to scale it on bigger hardware.

I believe that scheduler overhead can't be ruled out (not for python nor any other program) on a server since we've sometimes observed that the scheduler could be the bottleneck under some circumstances. For instance, some Linux schedulers used to show poor perfs when using nested cgroups with resources quota enabled.

Also, I'd like to state my first point again: you need to see how the number of workers will influence the memory usage on your system. Especially with python, if you've got a lot of workers, you can expect some memory fragmentation that can impact the perf of your system.

There is also the not much discussed issue of having shared resources between all of these threads and the impact of such a threading model on the engineering part if writing such a program. I personally haven’t seen the thread per connection model in a successful large scale server.

I think the Rachel by the Bay blog post does a good job of explaining what event loops are doing under the hood, and how that can lead to bad tail latency for web requests.

One other thing I noticed is that this uses aiopg for the async db queries instead of asyncpg, which is more widely adopted and IMO much better.

I was hoping to re-run these benchmarks myself with asyncpg.

Looks like actually running the benchmarks would take a bunch of manual work. In fact I don’t see instructions for running these benchmarks to replicate your results.

That's rude.

Let's put it this way. NodeJS and nginx leveled the playing field. It destroyed the lamp stack and made async the standard way of handling high loads of IO. From that alone it should indicate to you that there is something very wrong with how you're thinking about things.

You know the theory of asyncio? Let me restate it for you: If coroutines are basically the SAME thing as routines but with the extra ability to allow tasks to be done in parallel with IO then what does that mean?

It means that 5 async workers in theory should be more performant than 5 sync workers FOR highly concurrent IO tasks.

The logic is inescapable.

So what does it mean, if you run tests and see that 5 async workers are NOT more performant than 5 sync workers ON PYTHON exclusively? The theory of asyncio makes perfect logical sense right? So what is logically the problem here?

The problem IS PYTHON. That's a theorem derived logically. No need for evidence or data driven techniques.

There's this idea that data drives the world and you need evidence to back everything up. How many data points do you need to prove 1 + 1 = 2? Put that in your calculator 200 times and you got 200 data points. Boom data driven buzzword. That's what you're asking from me btw. A benchmark, a datapoint to prove what is already logical. Then you hilariously decided to dismiss it before i even presented it.

Look, I say what I say not from evidence, but from logic. I can derive certain issues about the system from logic. You just follow the logic I gave you above and tell me where it went wrong and why do I need some dumb data point to prove 1+1=2 to you?

There is NOTHING made up above. It is pure logic derived from the assumption of what AsyncIO is doing.

>But note how I said "probably" because I don't know for sure, and I'm not comfortable with making things up and stating them as facts.

But you seem perfectly comfortable in being rude and accusing me of making stuff up. I'm not comfortable in going around the internet and trashing other peoples theories with accusations that they are making shit up. If you disagree say it, I respect that. I don't respect the part where you're saying I'm making stuff up.