Ask HN: Are you doing async programming with Python?
If yes, what are you building, and what’s your opinion of async Python?
Open a python REPL, type:
>>> import this
There should be one—and preferably only one—obvious way to do it
…
It has not been true for years. Python 3 especially, took a sledgehammer to that ole poem.
See http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... which is about this effect. I am reminded of that blog post every single time I work with async in Python :/
async in Python 3 goes from "there's lots of solutions, which should I choose?" to "there's an obvious solution in standard library (unless you really want something else for a very specific reason)"
from __future__ import braces
and
import antigravity
> There should be one—and preferably only one—obvious way to do it
I feel this can be interpreted many ways. Sure, async means there is more than one way you can do requests to get web data now, and more than one way you should structure your app - but not every app needs to do API calls async, and sometimes the complexity is not worth it.
Other times, yes, of course you need to do requests async. They're taking three minutes each and you have to do twenty of them.
But I'm not sure if I have the same opinion of asyncio. To be honest, I don't even know if I understand it completely. There are so many concepts and IMHO they're not really intuitive, so it's difficult to put pieces together and find out how those individual concepts relate to each other or work with each other.
On the surface it looks simple, but it's really like going down a rabbit hole. The more I try to understand it, the more I agree with Armin Ronacher's article on asyncio [1].
[1]: http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio...
There was a post on HN that was asking about real-world examples of asyncio code that wasn't `asncio.sleep()`. I think my project is pretty good example for that - and a strange example too because it involved a lot of hardware interfaces.
I've been meaning to open-source the project ever since we axed it. I was the only real programmer on the job, so I'm itching for feedback on it. So if anyone's interested I'll post it here.
I had actually written it 3 times over. First as raw socket server, then in twisted, then on top of asyncio.
So yes, asyncio is a bit of a mess. It's documentation is a bit odd and I have several gripes. What helped we was realising that there was an 'async way' and an 'callback way' to everything in asyncio. So not mixing the two together and ignoring the callback parts really _really_ helped.
I don't know if this is true, but I recall reading somewhere that asyncio was not designed with coroutines initially and this explains the dual-personality of the library. I can't find any evidence that asyncio existed before 3.3, so I can't back that up.
But even then it's hard to debug, or work out ideas in the REPL, testing is a bit more complex than it should have to be, etc. There are probably better languages to reach for (I'd love to see rust firm up a one true way of async).
Maybe Python 4 will mark a maturing of asyncio?
Note that our use case might be different to many in this thread - in contrast to HTTP, we write mainly event-driven message-oriented applications. It is often practical to have e.g. multiple different consumers within one process - without async, we'd need to resort to threads which brings a plethora of new problems. Async Python does offer a good solution for this.
That said, there are a lot of places where async could be improved.
Language-wise it's pretty okay, but the asyncio library contains too many concepts (some perhaps introduced historically and now deprecated) and the docs aren't doing very good job explaining those. IO loop management could be simplified and some of the related functions should have different, better names. Error messages and stack traces could also be better when async is involved. As far as testing goes, pytest-asyncio could also enjoy some love.
But overall, no major complaints. Few catches aside, it works pretty much the same as in C# or JavaScript syntax-wise, which is what you would expect from this approach to asynchronous programming.
I did some small contributions for HA components and didn't have to get deep into async just yet, but my general feeling is that it is less 'natural' that async programming in a language like Go. But Go has the bigger advantage is was born with async in mind, so it doesn't lag behind that much with 3rd party libraries support like Python does.
Because both can be solved in a sync server via thread pools.
My opinion is -like somebody said- it breaks the Zen of Python. When I started using it I had doubts about moving to a more async 'friendly' language. But I decided to stick with my old friend Python... even when I don't recognize him very much after some changes after version 3.
It's pretty easy to get started with and with the new keywords simple enough to teach. The testing story is a bit horrible due to how coroutines work though, having to create asynchronous True and False when patching returns for example is a pain.
Can not really tell what I am working on, but so far I am more or less happy with it.
It is not as simple as advertised though ("just use async/await"). In order to write stable, performant applications you have to understand more or less the underlying mechanisms of asyncio (what are tasks/futures, how does their lifecycle work). Then there are quite a few things which work not as expected and you have to understand them as well (e.g. exception handling). If you get into the nitty-gritty details you will have a good experience with asyncio. If you don't you better stay away from it or you will have a hard time.
Performance wise it seems pretty good so far btw.. Did not really hit its limit yet though.
I used it to write an arbitrage trading bot. I learned python and asyncio just for this, and it is working as intended.
Asyncio with aiohttp is used to capture price data from 4 websockets and handle requests to exchanges via their APIs.
Python was easy to learn with “learning python the hard way”, asyncio was also easy to learn thanks to lots of available material online, and it looked less confusing than threading.
Again, speaking as a beginner, I quite like asyncio because to me it looks just like synchronous code. To me it’s easier to think about and keep track of the control flow and shared data than with threading.
Asyncio looked like the easiest and fastest way to accomplish my goal, and it worked.
One was an in-house DB abstraction/API that handled multiple DB hosts, read/write query routing, connection pooling, query queueing, metrics, and more. The other was a link forwarding service that took encoded links, decoded and tracked them and their metadata, and forwarded the user.
It's okay, it can be a a little confusing at first but worth the time investment required to understand it. The API has been improved with all of the recent 3.x dot releases. The new run command in 3.7 should make it a lot easier for folks to get started.
So the first step is to write a crawler and async is a really fast way to do that because everything is network IO bounded.
Does it use 'asyncio'? Nope.
I don't have my hopes high for this one though.
Python is awesome for a lot of things but go really shines for async style programs.