Asynchronous Programming in Python: Asyncio
djangostars.com
djangostars.com
I've been using Python for more than 12 years now. Used threads, then Twisted, then eventlet, then gevent.
Switching to Twisted was hard but oh well, there was nothing else that did the job right for lots of concurrent requests. Unfortunately it also created a separate library system, fragmented and cumbersome. You'd want a library, you find it "oh but it doesn't return deferreds it blocks the current thread". Or vice versa when using threads, I'd find the perfect library to parse my protocol but it returned deferreds only. Then eventlet and gevent came about it was like a breath of fresh air. I remember refactoring a largish code base from Twisted to eventlet and it was something like a 50% reduction in code size. Then gevent came about with more improvements and so on. Then async came, and now it's like back to Twisted. Find a library but well, it return the async thing not a result, so it has to live in a separate ecosystem. You have write async this async that everywhere.
Also, maybe if async came 5-7 years ago it would have taken over but now once people have to rewrite everything with async they'd also start looking at other languages: like Elixir and Go which somehow managed to avoid "async everywhere" and still do the right thing. Python almost had that too with greenlet based libraries. Embracing that would have probably made more sense, but oh well...
1. fork'ing just doesn't work properly; something about the event loop in the child hangs.
2. You can't easily spawn an honest to god hardware thread anymore. (Since the typical call is monkey patched!) Yes, you can get the original threading module by getting it out of gevents hands, but…
3. …it doesn't compose well. Libraries, in order for gevent to play nice, need special "if gevent then …" sections.
#3 hasn't been an issue for us, monkey patching works well. We've switched to pure python versions of a few libraries. When not available use a pool of instances.
Gevent explicitly says :
"Given that epoll is more sophisticated than select or poll (e.g., support for edge triggered and level triggered events), I don't know if we can emulate that close enough with the capabilities that libev exposes (we'd need to pass the stdlib tests when monkey-patched), and I don't know what the performance characteristics would be (and if the subtle differences would break apps) "
I don't see a path forward for gevent with these issues in place.
First, forking is "firing up a few processes". Yes, you can do that prior to launching your process, and then let gevent just have the process, but you have to be able to do that, and it just doesn't make sense in all contexts. This is what I meant by gevent fights composability.
> or threads
There's a limit to what a single CPU can do, and eventually multiple hardware threads can buy you actual tangible results.
> Yeah there's limitations when CPU bound, but that's python.
My point being that it's very difficult to get to CPU bound when your greenlet library prevents you from taking advantage of threads or processes. "but that's python" implies to me that you think Python is hamstrung to a single thread, and that's not the case: Python is quite capable of not only hardware threads, but performing computation on hardware threads in parallel, simply because not all of my code is in Python. Sometimes, a component knows it needs an honest-to-god HW thread, and the monkey patching fights that.
HW threads additionally lets you run multiple-long running computations; even if they're mutually bound to a single CPU, preemptive multitasking at the OS level (and Python's interpreter occasionally releasing the GIL) will ensure they all get time. In a cooperative greenlet, if you do not yield, you starve the other threads. A library making a long running computation again needs intimate knowledge that at the higher level, greenlets are in use: this breaks layers and mixes responsibilities (greenlets) in where it doesn't belong (some library calculating a thing). Greenlets do not compose.
> We've switched to pure python versions of a few libraries.
And you pretty much have to; if the library called out to C, it would block. But this fights good engineering practices: you can't just implement your library once in the language of choice, and then bind it into another language.
> When not available use a pool of instances.
This depends on what the library does; a "pool" does not necessarily save you. For example, if the library blocks, you need a threadpool (a real one) to process inputs to it.
Libraries exposing an event loop, perhaps with some basic primitives, have a hope of communicating across things like language barriers.
We communicate across languages quite a bit using sockets. A given system may have 20 processes that communicate with each other and the outside world. Most but not all of the python processes use gevent. Composing software with sockets scales well and avoids compatibility issues. It's rare for our python processes to be cpu (or core) bound, but it's easy to be IO bound and that's what gevent solves.
For example, would it be a bad idea to put my email module on an async function as opposed to in a different application?
There's no 100% correct way to say when to use one or the other, but here's one way I would think of them.
Microservices are used to help define the architecture of your system overall. This is more ... high level. Async would be more at a process level.
So, for your example, you could have a flask microservice that exposed an interface that accepted a list of email addresses and a message to send to each. The request handler for that could then call a method using asyncio to send the emails. This would allow the microservice to remain responsive and wait for more requests.
Yes, it is easy to understand (and even more than that - yes it can be in its own module/blueprint anyway). My previous thinking has been that if I have a message queue and separate application then I get the benefits of failure reporting through the queue as opposed to potentially missing something if I get a failure through async. Although I guess they aren't insurmountable.
Then, inevitably, you have growth so that one email function it may begin with, but later you're doing other fancy things and even though you may just want to send a single email at the start, later you're making database calls and doing other bits and pieces to make the template more fancy and personalised etc. So, better to start the architecture from the start in a slightly more complicated fashion..?
No. Start with "bad" architecture and refactor as needed.
Another layer of abstraction solves any problem. If you don't have a problem yet, then abstraction is unnecessary complexity.
BTW, writing your own async code may actually be more complicated than just having a message queue and an email program. It depends on how much trouble your message queue is giving you. Can you use a Sqlite table instead of a "real" message queue?
I'd have the web-server append "messages" to a sqlite table and have the emailer read from the same table. You can use cron to run the emailer every X minutes. If the table gets huge, use cron to run an archiver that prunes old messages during low-usage hours. Sure, the pruning will temporarily lock the database. If that gets to be an issue, switch from SQLite to PostgreSQL. If that architecture starts to fail, that's a good problem to have.
You could argue that writing to the queue should be done asynchronously inside the web-server, but, again, wait until your customers complain that the server isn't responding quickly enough.
One big reason for this is because we're often wrong about what problems we'll have in the future, or in what way they will manifest. Moreover, often the "bad" solution will work for far longer than we expect, and the day that we're anticipating where we'll be having problem X might never even come.
What's worse is that in the meantime each layer of abstraction adds a tax to how easy it is to fully understand and work with the code, to change things and add new things.
What you do want to do is to write code that is /open to/ being abstracted. E.g. keep things structured nicely, start with a basic function (named similarly) or a few related functions before putting in a full blown class, name things well so it's easy to find them while refactoring, etc.
I've done this with iteration 2 (current version) or my backend but now I'm not a one man shop and have a team (well one other junior backend and a senior mobile). And we are architecting for some pretty major new features so it's interesting trying to strike a balance with both what I've learned along the way and what it seems 'best practice' is when it comes to python
The pattern I described of using SQLite tables to mimic a message queue will be very easy to switch to PostgreSQL or MySQL. If you hit serious scale you can switch to something like Kafka or Kinesis. That'll be a bit more involved, but hopefully your revenue and team scaled to match the compute needs. When you switch to Kafka, you can use Spark or Storm instead of cron. If you switch to Kinesis, you might choose AWS Lambda. So you're not writing yourself into a corner.
Personal opinion: It's not needed if you don't want to code fancy. In some regards its even disappointing because the name promises to solve a generation old Python problem, which it actually doesn't even tackle.
There are already two fairly large ecosystems around asnycio network libs + http frameworks which are large rewrites of existing libraries (https://github.com/klen/muffin and https://github.com/aio-libs). It seems like having a whole framework/ecosystem for loop lifecycle management, async libs, state/comm, etc will unfortunately be needed. I hope we find a way to work around this although I doubt there is a good way.
... twisted has been around for ages for exactly this reason as a 'framework' unfortunately this means it's not 100% ported to python3 yet, but I suspect (not a dev, etc) that as twisted becomes more mature and 2.x dies off, there might be more overlap between these two things.. and perhaps other similar projects
A coroutine gets executed differently from an ordinary function (or an ordinary generator, for that matter); when you call it using the await syntax, the Python interpreter can choose to run some other code while it is waiting for your function to return a result from I/O (for example, a network request). A normal function call doesn't allow that. (A generator might, if you use yield from and do a number of other things, but you have to do them by hand, whereas the await syntax does it all for you automatically inside the interpreter.)
This is a great talk where the presenter goes into how asyncio works in detail and builds a mini version of the asyncio library:
As I understand it, async functions have an extra flag that ordinary generators don't have, and the interpreter does some different things if that flag is present. But I have not dug deeply into the source code.
def grep(pattern):
print "Looking for %s" % pattern
while True:
line = (yield)
if pattern in line:
print line,
def grep(pattern, line):
if pattern in line:
print line,
g = partial(grep, pattern="python")
g(line='python in a line')
EDIT: As I see more examples of coroutines especially the pipelining use cases, I was wrong and agree that it's completely different from partial functions.
What are the most confusing aspects of using async/await? Is it the implementation (i.e the syntax in JS, C# or Python) or the theory behind it?
This is one of the places where the Zen of Python comes to mind:
> Explicit is better than implicit.
Edit: To see what I mean, compare Microsoft's async/await example to Python's:
https://msdn.microsoft.com/en-us/library/mt674882.aspx
https://docs.python.org/3/library/asyncio-dev.html
The Python example tells you how to set up an event loop and add awaitables to it, while Microsoft's "complete example" just has you add async event handlers to a window without showing you anything about setting up an event loop. It wasn't until I saw the event loop being set up in Python that I finally understood what async/await did.
Python asyncio library supports coroutines, callbacks, futures, thread-pool executors... All have similar APIs and dissimilar semantics. For instance, canceling a coroutine is different than canceling a future and different than canceling a thread. (I've heard asynchronous exceptions are bad, but this is just an example.)
Error handling is difficult. You can't observe an error if you don't remember to write code to observe an error.
Also confusing is that coroutines in Python can be driven by any event loop. Which is great! You can write your own. But that will be even more confusing. Erlang has one OTP, Python will have many.
Exactly. Python had a multiple thread and blocking I/O model. That's well understood. Javascript had a single thread callback-based model. That's well understood. When you have both concepts in the same system, things get much more complicated. It's even worse when it's a retrofit.
Consider Python's Queue module. If you call Queue.get(), your thread can block. But the async system doesn't understand that kind of block. (Is that right?) So you've blocked all the async stuff running off that thread.
Go has lightweight threads too, called goroutines. But because it wasn't a retrofit, all lock mechanisms are compatible with lightweight threads. When a goroutine blocks on a lock, its underlying thread is re-dispatched. Python 3.6 isn't set up that way, because async operations were retrofitted in parallel with the existing machinery.
I understand the idea of blocking I/O and skipping that until a function/thread returns.
Of course, many high-performance, I/O intensive applications like web-servers can effectively make use of a combination of multiple threads and I/O demultiplexing as well.
The biggest practical difference is synchronization-free programming. In other words event loops are already always synchronized and you can do whatever you want with shared memory knowing that no other code is going to interfere until you explicitly stop and give up control. This allows for more reliable and more complex concurrent applications with less effort.
You understand it better than you realize. The idea of a "thread" is that it's a sequentially run context for computation with a bunch of context carried with it (a stack, references to a heap). This sequence of computations can be pre-empted.
When OO systems offer "promises" what they're doing is essentially freeing a thread from the continuity of the computation and leveraging closures or object scope to hold the context. Instead of a thread representing the computation, now threads pick up computations and work on them until they stop for I/O, then resume them when the underlying async I/O layer has work for them.
I can tell you about it in terms of monadic computations or how it relates to other concurrency strategies if you'd like, but it will become more theory-dense rather quickly.
Only C (C++, etc.) and Java (C#, etc.) families of languages that share memory between threads can scale better with async. on multicore processors when it comes to network related performance.
Edit: Please comment if you downvote.
Edit: Now I can't comment, the site tells me I'm writing too fast.
You can prove this to yourself via construction: you can write an interpreter that simulates threads (and even pre-emption) with Promises, Actors, or a more formal monadic style. You can repeat this feat for any single member of the set and if you do it carefully that's a proof by construction.
Related reading, somewhat famous: https://pdfs.semanticscholar.org/2948/a0d014852ba47dd115fcc7...
Essentially what you choose to implement is arbitrary and should be informed by your underlying performance requirements coupled with your preferences.
And, as people have noted: your post seems confused. Everyone benefits somewhat uniformly from asynchronous operations. Java's implementations are often a little bit more tunable because the programmer can provide the threadpool for async workers.
Of all of the languages you named, Erlang is the most distinct as it offers an abstraction (and cost) of 1 heap per actor from the perspective of the garbage collector. This decision (made in a time when GC algorithms were quite a bit worse than they evolved to be) lead Erlang to something they want to keep: an abstraction that makes remote and local computation identical. Other languages that follow in Erlang's footsteps (Pony comes to mind, yay Pony!) do not do things this way.
[1]: Okay EitherT ContT. Sure. Get picky.
P.S., You cannot comment because your overall karma is too low, putting you in a rate-limited category. It generally clears up in a few hours.
However, I do not think there is much more of value to say other than to thank that one poster for a few pretty cool papers!
Here you can read about how async. works in my system: https://github.com/tinspin/rupy/wiki/Fuse
I might not be stroking the HN community in the right direction but trust me, I'm not confused.
> trust me, I'm not confused
Sorry but you really are confused about this. You said that Python, Go and Ruby don't have shared memory between threads, but they simply do. All of these languages's concurrency and parallelism models are fundamentally shared-memory, and all even allow completely unsynchronised access to shared memory if you wanted to do that.
If you won't believe me on that, here are five peer-reviewed papers from reputable venues on Ruby alone that talk about the shared memory model it has. Either all of these experts and all the reviewers are mistaken, or you are.
B. Daloze, S. Marr, D. Bonetta, H. Mössenböck. Efficient and Thread-Safe Objects for Dynamically-Typed Languages. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA), 2016.
C. Ding, B. Gernhardt, P. Li, and M. Hertz. Safe Parallel Programming in an Interpreted Language. In Proceedings of the First Workshop on the High Performance Scripting Languages, 2015.
L. Lu, W. Ji, and M. L. Scott. Dynamic Enforcement of Determinism in a Parallel Scripting Language. In Proceedings of the 35th Conference on Programming Language Design and Implementation (PLDI), 2014.
R. Odaira, J. G. Castanos, and H. Tomari. Eliminating Global Interpreter Locks in Ruby through Hardware Transactional Memory. In Proceedings of the 19th Symposium on Principles and Practice of Parallel Programming (PPoPP), 2014.
W. Ji, L. Lu, and M. L. Scott. TARDIS: Task-level Access Race Detection by Intersecting Sets. In Proceedings of the 4th Workshop on Determinism and Correctness in Parallel Programming (WoDet), 2013.
Well thanks for this, can't wait to sit down & read it.
They might try to work around that, but they will never be able to run one common task to/from many sockets on all cores of one machine at the same time.
Python has a GIL, so only one thread can interact with the Python interpreter at one time. You can have any number of threads running at one time if they release the GIL and don't touch the interpreter (i.e if they are doing IO or calling a C function).
They, however, are still real OS threads, running (waiting for a lock) on separate cores and sharing process memory. That's the definition of a thread. They are not fake threads. They share memory. They run on separate cores (perhaps).
I'm not sure what your last sentence means.
I'm not sure why you're trying to salvage this or what your endgame is here but please reconsider.
That is certainly more accurate, but seemed needlessly confrontational in my first draft, so I opted for the more generic phrase "confused post", which implied maybe the post itself was simply jumbled up and didn't clearly offer your intent to the audience. You did ask after all.
I was trying to give a non-confrontational way to explain why I almost downvoted your post for being very misleading, and opted instead to respond with a slightly more correct post.
HN needs less fighting and more knowledge.
I thought what you were trying to say was, "parallelism is not the same as concurrency" but that it got jumbled up. That is a correct statement.
JavaScript doesn't have threads so the point is moot.
Only Erlang doesn't share memory between processes.
Nim doesn't share memory either.
In Erlang and Scala you can solve it even more ellegantly using actors and chanels. Go has goroutines and channels. All of these can span multiple cores.
There's also a collection of resources here: https://github.com/elazar/asynchronous-php