Python: Generators, Coroutines, Native Coroutines and Async/Await | Hacker News Reader

Python: Generators, Coroutines, Native Coroutines and Async/Await

masnun.com

147 pointssnw30 comments

If you don't have the latest asyncio version, change `ensure_future` to `async`.

> asyncio.async(display_date(1, loop))

See this https://github.com/python/asyncio/pull/242

The author is using the latest release of asyncio, will be good to mention that.

Anyone know when most of the python stdlib (the batteries included part) will start itself using async/await (for any/all things that could block)?? I suppose it will never change due to backwards compat. (which feels odd...)?

More than that, believe it or not, but synchronicity and blocking code is considered a feature: it's simple and straighforward.

Most code doesn't need to be non blocking. We only hear so much about non blocking because the Web is the big thing and Web programming benefits from it.

But there are hundreds of other fields in programming where IO is not the main issue, and hence don't need the additional complexity of being async.

Granted, asyncio does a good job at making async easier, but imperative code will always be easier to use, learn and teach. These qualities are very important, and made Python famous : easy to grow with, and yet powerful.

So the stdlib will remain as-is, the useful, easy and beautiful thing it has been.

We may see additions made to it so you can ALSO, when needed, do more stuff async. But really, I think we will see more and more external lib just doing that, so for when you DO need async, you will just pip install.

Things I do hope will have an additional async API though:

    - urllib;
    - sqlite3;
    - subprocess.

You know what I've always wanted to see (and it's quite possible one exists; I have never actively searched for one)? A completely asynchronous programming language. Every single operation is carried out asynchronously; everything from addition to while loops.

Synchronization would be a nightmare, but I think it would be a fun exercise to see what you could do in that framework.

Erlang is sort of this. All I/O is done by passing messages to Port objects which send the results/errors back as messages. Your process doesn't block until you tell it to and you don't block on a single operation exactly: you block on waiting for a message in your process's mailbox.

The nice thing about this approach is that you can emulate blocking or non-blocking I/O within the same function or even against the same file descriptor. And processes are so cheap that if your callees do block in unexpected ways you can just move that one call into its own process that just sends you a message when it's done.

Such a language would translate really nicely to hardware. Funnily enough, verilog / VHDL are the closest I can think of...

You don't get what you would normally think of as "while loops" though, which makes sense if you think about it.

Check out MyHDL and Cocotb:

http://www.myhdl.org/

http://potential.ventures/cocotb/

Hoare's CSP is exactly this. It's not a programming language per se, but I know of several DSL's that were built using CSP syntax. https://en.wikipedia.org/wiki/Communicating_sequential_proce...

How would that work? My understanding of async is that it's a way of interacting with IO. So how would the examples you give (additional, while loops) behave asynchronously, since they don't involve IO? (Except for maybe memory IO..?)

Addition still takes time though doesn't it? The idea is I could start an addition operation, and then be notified in some way when it's done.

I think the idea most similar to what the grandparent wants is instruction-level dataflow, where programs are DAGs, and instructions are only run when their inputs are ready, rather than when a program counter reaches them.

Physical dataflow machines were built that did this in hardware.

> programs are DAGs, and instructions are only run when their inputs are ready, rather than when a program counter reaches them.

At a low level, that's exactly what modern CPUs do.

https://en.wikipedia.org/wiki/Out-of-order_execution

Right, but only on a single core and only over a small sliding window. Real data flow machines made it explicit rather than a fragile optimisation and scaled it up.

True, though with multiple functional units there is some parallelism. Just thought it was an interesting comparison.

A program as a DAG.... Whoa. That's an interesting concept to try to wrap your mind around.

Just fyi, a library openstack (and myself and others) has created (in python) basically lets u program your workflow (really dataflow) as a DAG, and then the library will run it reliably (and in parallel and so-on). I'm more than willing to answer questions about how it does that if people are interested...

http://docs.openstack.org/developer/taskflow/

https://pypi.python.org/pypi/taskflow

And yes u can do things like built equation solvers to:

http://docs.openstack.org/developer/taskflow/examples.html#l...

And other neat things:

http://docs.openstack.org/developer/taskflow/#examples

That's exactly how Haskell's IO monad works (albeit a little simplified)

This is called Continuation Passing Style: https://en.wikipedia.org/wiki/Continuation-passing_style

Nothing about CPS ensures asynchrony.

But it enables it.

This was attempted with Midori, see here:

http://joeduffyblog.com/2015/11/19/asynchronous-everything/

There are some doing that by default, but I can't remember the names. Every operation can be done in parallel unless you declare a necessary order for some subset(s) of operations.

Is lazy evaluation asynchronous enough for you? It's more predictable than random asynchronous I/O, though, so synchronization is easier.

Haskell does this.

sqlite3 is unlikely because it's only a small bridge to sqlite's own API which is blocking

Still, it's possible to write blocking/nonblocking bridges for that stuff

True. Yet a standard async DP API + wrapper around a process pool to integrate with it would be nice so ORMs move on. Righ now we are stuck with the status quo and all asyncio adapters for peewee / SQLAlchemy reinvent the will by creating some kind of pool and an ugly syntax to write each call.

The cost of such a change also compares badly with being able to wrap anything you want so easily.

You can't just sprinkle async onto existing functions and have the awaiting code resume magically: Going async also requires an event loop.

Adding an event loop to an environment where there previously has not been one is non-trivial.

Also, changing existing blocking methods to be non-blocking would break everything that uses (and expects) those APIs to be blocking.

I think it's a combination of impracticality and lack of motivation that will keep the stdlib largely blocking.

* File IO (`open()`) will remain blocking because there's no practical way to implement it in asyncio. This isn't Python's fault, the OS support is a mess. * Async Socket IO is already implemented in the stdlib. * The stdlib HTTP (urllib, urllib2) is pretty awful and I don't know of anybody who uses it, instead they opt for Requests. aiohttp is the asyncio equivalent of Requests. * SMTP would be nice to see async. * SQLite relies on file IO, so is likewise doomed to always block.

I can't think of anything else in the stdlib that is IO-driven.

Edit: sametmax mentioned subprocess, which I had overlooked. That would be a nice one to see.

Async is not new. life has been doing it for millennia. Your heart does not stop beating while you breathe, yet both systems are interrelated. The body accomplishes this through the use of 'buffers'. Pause one system long enough and the other will fail. So we have evolved a 'scheduler' - the hindbrain that starts to scream louder and louder when the event queue of one system becomes too long. It might be instructive to model this in silicon. Have one cpu core be the scheduler. Have every set of interrelated processes (that are Async) register their variables with the scheduler. The scheduler then keeps track of these sets. When a variable in a set changes, pause that process for two cycles.

A good article, but it should mention async network programming (probably with http://www.tornadoweb.org/) which is the major driving use case behind the adoption of async programming patterns.

Reply on news.ycombinator.com