Python Coroutines with Async and Await | Hacker News Reader

Python Coroutines with Async and Await

lwn.net

The async/await model gaining traction makes me really sad. I use it all the time and it just seems like a giant hack, a band-aid to hide "scary complexity" from the developers. A typical Microsoft approach to things, but it should stay there.

First issue - the necessity for at least two of each method: "DoSomething" and "DoSomethingAsync". Weak. Now as a library developer, you just doubled your work. Sure, sometimes one method could wrap the other. But it's still weak sauce.

I much prefer the Go model - you only code "sync" API's, and let the consumer of your API wrap it in a go-routine and make it "async" if they wish. Much simpler, better, composable.

Async/await is also gaining traction in Javascript. Really, really sad we still try to avoid concurrent programming and how to do it right vs easy.

It's trivial to wrap existing functions to make an async version.

    public Task<ReturnType> DoSomethingAsync(args) {
      return await Task.Run(() => DoSomething(args));
    }

In fact, it's so easy that I can think of at least two ways to automatically add async methods, either at compile time or at runtime. I would have to look into it a little more, but I think you could even write an extension method on object that would add the async version via reflection.

The upside of async/await is that you avoid callback hell. The code stays cleaner and is easier to read, both of which seem like wins to me.

That's a false choice you're presenting. "Callback hell" or "async/await". We have way more choices and opportunities.

Look at Go's CSP implementation, or Erlang's. No callback hell anywhere.

Right, but the languages that are looking at async/await weren't designed with that in mind. You could do a major overhaul of the language, or you could bolt on async/await. While Go/Erlang's way of doing things might be preferable to some, I'm not sure I see async/await as being bad enough to justify a major language overhaul.

With the world going multi-core more and more, the question that comes to my mind is - are the languages designed without concurrency in mind doomed to fail? It seems C++/C#/Java all look at concurrency/parallel programming as an afterthought - another library here, another library there, but none of them approach it as a fundamental programming necessity, on the same level as a basic loop or "if" statement.

Ultimately, it's possible to be semi-successful in designing scalable, concurrency-aware software with just after-thought libraries and language support, but is that really the way forward, long-term?

I disagree. What you're arguing, in effect, is that async/await is bad because you should handle asynchronous APIs by taking a synchronous API and running it in a different thread, which experience has generally shown to be a recipe for buggy code, inefficient code, or most likely both.

The underlying idea behind async/await is that synchronous methods shouldn't exist in the first place--you'd only have the asynchronous API. Async/await enhances that model by turning the promises' functional interface (which requires interesting binding to handle loops/recursion) back into an imperative control interface.

Not true from what I've seen Every C# library with support for async/await has synchronous versions of those same methods. In fact, Microsoft's own documentation recommends appending "Async" to the method name for precisely that reason - to distinguish an async method from a sync method.

Furthermore, "asynchronous" code doesn't need separate threads. A scheduler and green threads is more than enough.

Have you seen Go? You should take a look at it. It makes writing concurrent code basically trivial, using "green" threads and channels and a neat little "select" statement.

They only support both because most libraries still support older version of .NET

If they were to support only .NET 4.5 or higher, then you can just call any async method in a sync manner simply by doing httpClient.GetAsync(...).Result

async/await in C# is a beautiful thing.

I like the idea of a way to make async code more readable but the execution of it in .net is very awkward, every async task is magically run on some threadpool and when the task is done your async function will also continue on that second thread. This means that code above the await and code below it doesn't run on the same thread, and there is no way to control it, this is just asking for trouble if you have other shared state. Shared state and threading is of course always trouble but with the async Task-library you hide it away, with explicit callbacks its easier to see where the entry points of second threads are and you can enclose them in critical sections.

However, in javascript the idea works because javascript is single threaded so you can be sure that the await-"callback" is always run when nothing else is. Explicitly defining some kind of message-pump that you can control the await-continuation yourself would be way around this in multi threaded languages but i haven't found a way to do so in c#.

The Python standard library comes with support for a lot of common network protocols - TLS, HTTP, SMTP, etc.

I'm wondering how this will be handled in async. Will everything need to be reimplemented as separate async implementations, or is there a way to have a common implementation that supports both sync and async? Will e.g. async HTTP be added to the standard library?

I think this was a topic of discussion when they first added asyncio, but I never saw an official decision.

It's fascinating how async/await are spreading from C# to other languages like JavaScript and Python. As far as I know, no languages had anything like it before C# added it in ~2011.

What was the last truly-new language feature that became popular across other languages like this?

The idea of using coroutines for asynchronous programming is quite old actually. ML has used it since the 80s [1]. Python itself had it way before C# with Twisted's inlineCallbacks [2].

What C# (actually it was F#) introduced was the specific 'await' and 'async' syntax keywords. Those are being copied in other languages, and they're just sugar over coroutines.

[1] http://www.cambridge.org/gb/academic/subjects/computer-scien...

[2] http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-...

It's my understanding that this came out of the programming language research community and is just now trickling into production languages (just like generics did).

We're basically just getting the up to speed with ML from the 80's.

In python world, every networking programming trick trace back to Twisted.

Python 3.5 is turning out to be a version that people might actually be interested in.

I don't see why this should be a tipping point; this introduces no new capabilities, and a lot of hairy new syntax and magic methods.

Async operations were the one biggest hurdle when writing efficient I/O-bound programs. (Definitely, Python is not about efficient parallel CPU-bound programming, but it's fine for many I/O-bound tasks.)

Having used async & await extensively in C# they're pretty great... but I can't help thinking we're explicitly declaring something that mostly could be implicit if the language was designed right.

For example: if I'm calling an IO method that returns a value it should be able to figure that out and orchestrate the code to run asynchronously and await for the value when it's referenced. If I have several asynchronous calls methods in a method it should be able to optimize it so they all run in parallel as optimized as possible.

It might need some added declarative syntax on methods but I'd rather prefer that and make the compiler smarter, instead of offloading that complexity on the programmer per default. Sure you should be able to explicitly control asynchronicity if needed but in most cases that doesn't seem optimal nor necessary

Adding async/await all over your code adds to the "crud" that makes code more verbose and harder to read and understand

You need that "crud" so that you actually know what your code is doing. If the compiler automatically spins up another thread and finishes there while you return from that function without you knowing it would lead to all sorts of race conditions and other threading headaches. Making them explicit makes it clear what the code is doing.

that's not necessarily true, in some cases yes. But as I said, control could and should be there for the asking, I don't want a ORM style black box where god knows what happens behind the scenes... but I think you could declaratively markup methods so the compiler knows what is safe to do. For instance marking them as pure, mutex etc etc.

It's kinda the same thing except telling the compiler what to do you instruct it what's safe to do, and it can make smart decisions from that. If the compiler/environment detects any ambiguity or risk it might even query the programmer to declare his intent

> Having used async & await extensively in C# they're pretty great... but I can't help thinking we're explicitly declaring something that mostly could be implicit if the language was designed right.

Yes, basically, they are explicitly declaring dataflow dependencies. There are languages that handle this automatically, e.g., Oz [0].

OTOH, async/await is a solution that can be retrofitted onto an existing language that isn't fundamentally designed around dataflow dependencies without changing the semantics of existing code.

[0] as implemented in the Mozart Programming System, https://mozart.github.io/mozart-v1/doc-1.4.0/

The main purpose of the async/await keywords is to help the programmer reason about what can happen when.

Having calls be implicitly parallel makes it difficult for the programmer to predict what could be happening elsewhere in the program. E.g., this passage from the Oz documentation[1]:

"Oz 1, supports a fine-grained notion of concurrency where each statement can potentially be executed concurrently. This results in a fine-grained model similar to the actor model. A good exposition of the Oz 1 programming model is given in [Smo95]. Our experience using Oz 1 showed that this kind of model, while theoretically appealing, makes it very hard for the programmer to control the resources of his/her application. It is also very hard to debug programs and the object model becomes unnecessarily awkward."

For I/O in particular it is also difficult for the machine to know what is safe to do in parallel. For instance, two writes to the same file probably has to happen in order. What about writes to different files? What about HTTP requests? Most I/O has the potential to produce wrong results if rearranged in arbitrary order, and requiring the programmer to mark them sequential becomes a potential source of error and a cognitive burden.

In most cases it's better to be slow and correct by default, and allow the programmer to explicitly run things in parallel in the cases it matters.

[1] http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node1.h...

I agree that those can be issues... but that can be solved by annotating the methods to those constraints. The compiler can even highlight potential problems and ask that you annotate them.

The benefit of this approach is that the compiler should be able to see more opportunities for optimization and also result in fewer bugs.

This is pretty great - One feature I really liked from go was channels and the ability for goroutines to communicate with each other. Anyone know there a pep open for a similar idea in the python world?

You can get channel-like behavior using async/await and a custom event loop. I wrote a little example[1] that has a bare-bones implementation of this. I used it to translate the prime sieve example from Go[2] almost directly to Python. The code uses "message = await channel.receive()" to mimic Go's "message <- channel". Instead of using "go func()" to fire off a goroutine, I use loop.run(func()) to add the PEP492 coroutine to my simple event loop.

It's not an efficient implementation - it was really meant as a proof of concept that you can use async/await in your own code without any reference to asyncio.

[1] https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-si... https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-ch...

[2] https://golang.org/doc/play/sieve.go

relevant: http://lwn.net/Articles/644128/ ("PEP 492 vs. PEP 3152, new round")

It's linked towards the end of the article, after it's been explained what PEP 3152 is, so reading it after the main article is less confusing. :)

Coming from a Go mindset, having moved a significant portion of dev to Go from Python, in many ways for "goroutines", it seems as if there still isn't a parallel, unless I'm missing something.

In the simplest case in Go, it's easy for me to toss off some concurrent processing and not wait or care about what happens in those functions. For all intents and purposes, it feels like parallel execution (I know, I know).

I can't quite wrap my head around how generators or async/await gives me the same functionality. What am I missing?

Do you regularly set "GOMAXPROCS"? If not, you're just using coroutines in Go as well, just with less control over the scheduling of the execution.

I do. But it looks like I'm comparing apples and oranges. Goroutines and Erlang processes feel very much like lightweight threads. Async/Await still pause for return, but doesn't block the calling thread (it seems like). I'm trying to do some research on the patterns (mostly C# vs Go because this PEP is still newish) and it SEEMS like that's what's happening here.

Goroutines and C# async/await are quite similar in use and behavior, the implementations are very different though. Goroutines have their own stack but Tasks don't, they store values that cross await boundaries in a compiler-generated class and everything else uses the worker thread's stack.

Async/await doesn't feel like lightweight threads because you need to explicitly await tasks. You could invert the keyword usage to match Go (implicit await, explicit go) and end up with something like goroutines.

This is something that a Python-like language by the name of Nim already has. Interesting to see Python get it now too. http://nim-lang.org/docs/asyncdispatch.html

How would one await multiple coroutine/futures?

EDIT: await asyncio.gather(fut1, fut2)

Why not: async asyncio.wait([fut1, fut2])

The async keyword is used in coroutine declarations, so it wouldn't be valid syntax.

My bad... of course I wanted to write

await asyncio.wait([fut1, fut2])

Reply on news.ycombinator.com