Rust – Coroutines support merged | Hacker News Reader

Rust – Coroutines support merged

github.com

111 pointsadelarsq39 comments

This is experimental so far: https://github.com/rust-lang/rfcs/pull/2033#issuecomment-309...

> The key point then is that the experimental RFC is not enough to stabilize something. For that, we need a proper RFC that lays out the plan. This RFC can be informed by the experience and working implementation, so we should have a lot of data to use.

It's being merged into nightly (behind a flag) to gain understanding of how the feature would work. This isn't the day to celebrate Rust having fully-fledged co-routines coming soon to stable because there is no such guarantee yet.

Edit: removed extra word

This is the first "experimental RFC", so that's also kind of fun.

async/await is nice duct tape to integrate cooperative concurrency into a thread-based concurrency model but it has the drawback of creating an incompatible sub-language for functions within the parent language. Now every time you call a function, you have to check if it is a "coroutine"/ returns a promise or if it's a synchronous function.

This divides the language ecosystem and makes it hard for library writers to support both concurrency models.

Ask anyone who has done significant async/await coding in C#/Python. It gets messy fast.

Async/Await works a little better in JS because it always has had a cooperative concurrency model, standard library functions usually don't block (except sometimes in Node). Though it's still annoying to have to check whether a function returns a promise or not.

Languages that do this right are Haskell and Go and probably Elixer/Erlang. This usually requires eschewing LibC (which usually assumes a threaded concurrency model) and writing your own runtime.

Yes, this is all discussed in the eRFC (which admittedly isn't what is linked here, but is the first link in the description, https://github.com/rust-lang/rfcs/pull/2033) and various threads on internals before it.

> writing your own runtime.

This is not a cost that Rust can bear, and so that option is just not possible. Unfortunately, there's no such thing as a free lunch.

I wonder if Rust could support pluggable concurrency models (with the accompanying runtime support). That's what Haskell does and it seems to work. At the outset that seems like a better approach than creating an incompatible sub-language. The rust-facing side of the standard library already abstracts away many platform details.

Trying this was in fact one of the reasons that Rust ditched its runtime years ago; it doesn't work. Well, it didn't work for Rust at least.

The RFC that made this change https://github.com/rust-lang/rfcs/blob/master/text/0230-remo...

Interesting. The problems section isn't really convincing to me though.

Especially, indirect function calls for IO functions seem fine since IO is usually much slower than a function call. Also binary size increase may not be a problem for people writing web-scale servers.

It would be great if the Rust compiler infrastructure provided the necessary hooks for re-implementing std::/using a different concurrency model and independent projects could make those "problems" trade-off decisions on a individual basis. Rust-core could just only support/ship the native threaded model to lower maintenance burden for themselves.

> The problems section isn't really convincing to me though.

It was a big enough problem that the language was almost forked over it.

> It would be great if

Rust is low-level enough that you can do this, and some people have. But then people have to use your package. std is only special in that it can use unstable code but be part of stable Rust; otherwise, it's just regular old Rust code. The community overall abandoned all of those other things and is coalescing around Tokio, which could also be considered this, in a sense. Or Rayon, if you want data parallelism rather than async IO.

Would it be possible to make functions generic over coroutines/async?

One of the worst parts of async is there's an infectious duplication of pretty much all library code where there are separate sync/async versions of everything. See C# DoXxx() & DoXxxAsync() everywhere, where the only difference between the two implementations are an annotation and some keywords sprinkled throughout. If rust can do the same thing without doubling the code/api surface/documentation across all libraries that would allay the fears of many opponents.

You can do DoXxx() using DoXxxAsync() by calling wait() on the future it returns. No duplicated code necessary.

It's nice that that'll work but it's... kinda gross. Can it call wait() automatically via some auto type conversion? Ideally in a synchronous context it would call wait(), and in an async context it would automatically annotate await (and you could always override by calling wait() manually).

Haskell is doing this already with par and seq function to control parallel and lazy evaluation. I believe we should have something like this for Async

Wouldn't those trade offs just be picking a different language? I mean Rust is going for no runtime low level systems programming, if you add a runtime, then it brings what more to the table then say Haskell?

I approximately agree with you but there are lots of reasons I'd prefer Rust over Haskell. All data is thunked and boxed in Haskell, it's all heap-allocated and garbage collected, and polymorphism is through indirect pointers. Rust allows for "zero-cost" abstractions with a powerful memory-safe aware type system.

Erlang occupies a similar space.

Go is another option that's less like Haskell/Erlang but its type system is lacking/ inconsistent and it requires garbage collection.

So Rust has a real opportunity here that's not currently occupied by other languages.

Like a runtime only to manage concurrency, but still without a garbage collector and with unboxed types? Is that possible? I mean that's what the RFC looks like it wants to figure out.

Erlang occupies a similar space.

In what sense? I feel Erlang is at the other end of the spectrum from Rust. Can you explain further?

Erlang occupies a similar space compared to Haskell, in terms if it being a functional language with M:N green threads.

It is possible, there are many stack swapping libraries in C that don't use garbage collection/excessive heap allocations as proof (e.g. libpth). The RFC is trying to figure that out with rust, but their approach currently requires manually annotating functions with "async," creating an incompatible calling convention with existing Rust code.

I see, you want a functional language with first class concurrency, which also values high single thread performance and low memory footprint?

I don't know of many, I'm thinking Julia JoCaml or Red maybe a little.

but their approach currently requires manually annotating functions with "async," creating an incompatible calling convention with existing Rust code.

Can this be avoided? How else?

Not every data in Haskell is thunked and boxed. At least with GHC extensions.

Back in the day of Rust having a runtime, you couldn't call C functions, or functions potentially calling C functions, without worrying that it might indefinitely block one of the limited OS threads used by the runtime. With weirdly typed coroutines it's at least part of the type system.

I think Haskell happily spawns a new OS thread when one looks like it might be going to spend a while in C code?

same thing with Go as well.

Relevant RFC: https://github.com/rust-lang/rfcs/blob/master/text/2033-expe...

This looks like a huge win for Rust. The syntax seems to follow the C# and ES async/await syntaxes. With stuff like this, I can see Rust-based webdev becoming popular.

http://www.arewewebyet.org/

On a serious note, one of my middling priorities (unfortunately :( It always feels like you can't choose most your high priorities) is to build a rust-based website and a couple of other rust-based projects t get a feel for the language. It really dose feel like all the parts are there already and that as things get added, it'll just get easier and better.

Maybe try to find ways to bring rust into the middle of your higher priorities?

I'm in the same boat as the GP, in that I want to use Rust for a lot of things that I think it would be good at, but I don't get to choose my high priority tasks. And for me at least, my high priority tasks are not the tasks that benefit from Rust over other languages.

I could introduce it for the sake of my resume, or I could use the right tool. The former seems unprofessional and unproductive outside of jumping ship and leaving your co-workers with extra baggage.

High priorities being family, house repairs, house maintenance, and work. Nothing extraordinary, and I guess one could argue house upkeep is chosen.

I also have some personal programming projects I've been working on where I decided to value getting them out quickly vs learning a new language, and those are, all together, half done.

I've also been debating learning modern C++ to open potentially open some doors to a slightly different career than web programming (railway signalling, probably a pipedream at this point, but).

So, all in all, I love what rust represents, but it doesn't bring value right now. Once those personal projects wrap up my goal is to have a round of rust and C++ ones to do.

That site is a bit out of date; I should send some PRs.

"You can build stuff" is still a good summary, but lots of stuff has happened since this was last updated.

http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

Coroutines with "color" are problematic.

Sometimes, problems are inherent. Especially in a systems language.

There are already "Result" colored and "Future" colored functions right now. Async/await just adds syntactical sugar.

Implicit coroutines are also problematic: https://glyph.twistedmatrix.com/2014/02/unyielding.html.

As applied to Go: https://blog.mozilla.org/services/2014/03/12/sane-concurrenc...

Trying to wrap my head around this...as someone coming from .NET land, maybe someone can explain it in terms I understand? In F#, the default method of asynchronous programming is the async computation expression. That is, there is a type Async<'T> which represents a computation that can be executed to return a result of type 'T. The computation itself is "lazy" in the sense that it isn't hot at the time of creation, and can be executed multiple times to produce a result. This is in contrast to the C# way of doing things, in which a function has a return type of Task<'T> which is (under normal circumstances) kicked off at time of creation and can only be run a single time. Is this proposal advocating for the more F#-y way of doing things, or the more C#-y way of doing things?

C# has what you've described as well in the form of yield return.

It's largely the same thing to the compiler in that it needs to transform a method into a stack-less state machine. Whether or not the state machine is evaluated lazily on demand or concurrently is mostly irrelevant and just depends on the specific problem in question.

Here's a rendered version of the documentation added in that commit: https://github.com/Zoxc/rust/blob/a996d5eec70ba6733e23f2e56e...

This will be located at https://doc.rust-lang.org/nightly/unstable-book/language-fea... after the next nightly comes out in ~4 hours (I believe).

Looks exciting. I'm interested in how this might relate to the following use of generators from Python:

  def gen(*items):
      for x in items:
          if x > 5:
              yield x

  for y in gen(1,2,3,4,5,6,7,8,9):
      print(y)

Should print:

Does this new RFC support that kind of usage? I guess I'm not clear enough on how Futures work to figure it out myself.

Part of the RFC is the intent to use these to create iterators, so yes I believe that kind of usage would be covered.

Am I the only one who prefer epoll style concurrency than coroutines?

Are coroutines more somehow more powerful?

You will probably be interested in mio [1] then.

[1]: https://github.com/carllerche/mio

They're orthogonal? Coroutines as built here are just sugar around building the state machines you necessarily build by hand if you're using something like epoll. They can also be used to model state machines in general, so long as there's a well defined progression of states in one direction.

You would use these coroutines with tokio or mio, which are event driven frameworks that use epoll at their core on Linux.

Reply on news.ycombinator.com