The State of Async Rust: Runtimes
corrode.dev
corrode.dev
> An inconvenient truth about async Rust is that libraries still need to be written against individual runtimes.
That's really the heart of it. If it was really just a runtime, it wouldn't matter what implementation you plugged in.
...but it's not true for the rust runtime; I mean, it's understandable, how can you have one runtime that is multi-threaded and one runtime that is not, and expect to be able to seamlessly interchange them?
I understand it's hard and lot of work went into this, but let's face this. This article is right:
Practically speaking, tokio has become 'the' rust async runtime; but it's an opinionated runtime, that has a life cycle and direction outside of the core rust team.
That wasn't where we intended to end up, and it's not a good place for things to be. I, at least, agree: avoid async. Avoid teaching rust using async. When you need to use it, partition off the async components as best you can. I <3 rust and I use it a lot, but the async story stinks.
We should have an official runtime, officially managed, and guided by the same thoughts that guide the rest of the language.
What we have now is a circus. After 4 years of async being in stable.
If it goes into official runtime, then backwards compatibility will kill it eventually. You'll have situation where in year 2078 someone will ask why are we still having tokio when everyone is using telepathy lib?
> What we have now is a circus. After 4 years of async being in stable.
It's caused by strong backwards compatibility guarantees and long RFC process + unexpected problems.
Without strong backwards compatibility, no one would be using Rust.
RFC exists to hash out unexpected problems but so far we can't peer in the future.
Here is an example: Want to make Range from non-Copy to Copy. I.e. make a new type, rename old to new. That will be one year for RFC and two edition to stabilize circa Rust 2028.
By that measure async fixes have been blazingly fast.
This kind of situation happens and leads to a second, newer official runtime getting adopted and the older, legacy runtime being supported as long as is needed.
This happened with Java's official GUT toolkits which started off with the Abstract Windowing Toolkit (AWT) and then moved to Swing and almost moved to JavaFX.
Having multiple officially supported core components is not necessarily bad--it can be a sign of good backward compatibility balanced against the need to improve core components.
I think it is better than the alternative: multiple unofficial, de facto standard components that are incompatible. Who knows which direction each unofficial components will go and newcomers do not know which one to choose.
JavaFX was one of the many components that was spun out to the community and now lives as OpenJFX (https://openjfx.io/).
JavaFX would have replaced Swing had it not been for Oracle's change of direction.
Yes, but while Oracle employees may contribute to OpenJFX development, JavaFX is not an officially supported Oracle product (beyond legacy support of old JDK versions):
"Do I need a separate support contract for JavaFX?
No. JavaFX is part of the technologies covered under Oracle Java SE Subscription. As of JDK 7u6 JavaFX is included with the standard JDK and JRE bundles.
Note that for JDK 11 and later JavaFX is no longer included the JDK but remains available as a third party library from other vendors."
Source: https://www.oracle.com/java/technologies/faqs-jsp.html
What if Arc and Rc weren't in the standard library, and you had to import them via a crate, and multiple different (incompatible) implementations existed such that you couldn't use them at the same time?
Would that be ok?
How about Option and Result?
What if you could only use crates that used the same error library that you wanted to use?
What about boxing and custom allocators? Can you imagine if different crates could opt into different allocators and you couldn't safely drop an object without passing it back to the crate it came from because 'who knows' what might happen if you try to deallocate it using your allocator?
Should we not ship a default allocator and make that an optional thing too?
...
That isn't a language I want to use.
I'll take 'it comes with a default allocator' and that's good enough for me. If one day I get a stable 'you can pick, seamlessly at the top level, which allocator to use for your entire program', that's awesome!
...but it does not in any way mean, that I want a rust with no default allocator.
The default allocator is great. It works perfectly for most things most people need, and it 'just works', out of the box, the first time you use rust.
Async should work out of the box. It doesn't. That sucks.
This is somewhat of the case with many different Result/Error crates doing their own thing. So being in standard lib isn't a guarantee it won't fracture.
> What if Arc and Rc weren't in the standard library, and you had to import them via a crate, and multiple different (incompatible) implementations existed such that you couldn't use them at the same time?
First that's not currently what is happening in Rust. Second, I'd probably use the most popular and active one. Same as in JavaScript or Python.
I think my criteria for what is in the standard lib is following: How often does the domain change? And should it come out of the box?
E.g. are we inventing new ways to parse JSON? Yes ? Out of the standard lib you go. Is ARC/RC being reinvented? No? Go to standard lib.
Just make sure you have a firewall. Fortunately there is Linux software for just this need:
https://zapatopi.net/mindguard/
[JOKE]
Agree. This is why I refuse to use solar panels on roof. Science clearly tells sun is gonna burn down all its fuel and implode and at that point my solar roof will be useless deadweight.
I couldn't agree more. And my conclusion is, as it has been, to stay away from async until we have a sane situation.
Is it possible to avoid async with Rust when you use most common 3rd party libraries? such as ones to make API requests, database connectors, date/time, logging, deal with special kind of files etc.? or are we talking "the burden is on the user to set feature flags and carefully choose which crates they import into their projects"?
Is it possible to set up the Rust toolchain to not allow async in a project at all?
instead of reqwest you use ureq crate
> database connectors
can't answer at this time
> date/time
chrono crate has nothing to do with async
> logging
log crate with env_logger crate has nothing to do with async. pushing to something like elasticsearch instead of letting filebeat scrape your stdout is a different story
> deal with special kind of files etc.?
std::fs came first, the async stuff on top that recreate it in an async fashion came later. i'm pretty sure if you are dealing with a big file you can do std::fs with a "stream reader" basically
Asking because I could only find a category for explicitly async crates, not the other way around.. https://crates.io/categories/asynchronous
I just googled "synchronous rust http client" - the top result was a Stack Overflow question where the top (accepted) answer listed ureq as the first option.
(I'd still just use reqwest and Tokio though - practically speaking most of the concerns are non-issues in day-to-day work)
I actually did the opposite recently and replaced reqwest with ureq and managed to drop async and tokio altogether and greatly simplify my library. As a newcomer to Rust I kept getting pointed towards reqwest and tokio when ureq is far simpler.
Not using async or async crates is not recommended since most new or updated high quality crates now use async.
~> cd tmp\
~/tmp> cargo new futures-test
Created binary (application) `futures-test` package
~/tmp> cd futures-test
~/tmp/futures-test> cargo add futures-lite
Updating crates.io index
Adding futures-lite v1.13.0 to dependencies.
Features:
+ alloc
+ fastrand
+ futures-io
+ memchr
+ parking
+ std
+ waker-fn
~/tmp/futures-test> code .
~/tmp/futures-test> open src\main.rs
use futures_lite::future;
fn main() {
future::block_on(async {
println!("Hello world!");
})
}
~/tmp/futures-test> cargo run
Compiling futures-io v0.3.28
Compiling memchr v2.6.3
Compiling pin-project-lite v0.2.13
Compiling fastrand v1.9.0
Compiling waker-fn v1.1.1
Compiling parking v2.1.1
Compiling futures-core v0.3.28
Compiling futures-lite v1.13.0
Compiling futures-test v0.1.0 (C:\Users\steve\tmp\futures-test)
Finished dev [unoptimized + debuginfo] target(s) in 1.74s
Running `target\debug\futures-test.exe`
Hello world!
And like, to be clear, "the few hundred dependencies from tokio" is also misleading. A `cargo add tokio --features full` adds 43 dependencies to your Cargo.lock at the time of writing.
There's very much a shiny new thing problem in the rust ecosystem.
It's getting hard.
> Tokio's roots run deep within the ecosystem and it feels like for better or worse we're stuck with it.
Tokio has become a tentacle monster that is suffocating Rust.
Async is fine for webcrap, not good for embedded, and all wrong for multi-threaded game dev. The trouble is, the web back end industry is bigger than the other applications, and is driving Rust towards async. Partly because that's what the Javascript crowd knows.
Personally, I wish the web crowd would use Go. The libraries are better, the goroutine model, which is sort of like async but can block, is better for that, and garbage collection simplifies things. Rust is for hard problems that need to be engineered, where you need more safety than C++ but that level of control.
The "web crowd" leans towards async because most problems at scale where you would reach for Rust are almost always in a situation where they need to concurrently do a million tasks on 8 cpus. It's not because 'thats what the Javascript crowd knows', it's because, since the days of nginx (written in C), its been shown async i/o has better performance.
I don't see a lot of CRUD APIs in Rust - it's almost always database-like systems where the goroutine model and garbage collection cause a headache in terms of either memory usage or latency. I'm not sure if I agree that databases aren't "hard problems that need to be engineered".
That said, the reason Rust focuses so much on the web crowd, is because the majority of people paying the bills are the web companies. The Rust foundations biggest sponsors today are AWS, Google, Huawei, Meta and Microsoft (none of which I would describe as the "Javascript crowd"). AWS isn't hiring Rust engineers to work on game engines.
What I see more of is other industries just don't care about that much Rust.
embassy begs to differ
async/await is really just a syntax for building state machines in a way that resembles regular code. It's compiled down to the same code that you would write by hand anyway (early on it had some bloat in state size but I think it's all fixed now).
And embedded has a lot of state machines!
I had already replied to this article over on lobste.rs
https://lobste.rs/s/iovz9o/state_async_rust
The tl;dr is that I think this entire async concern stuff is ridiculously overblown. I suspect the vast majority of Rust devs, like myself, not only accept the current state as "fine" (couple things to work on) but are very happy with the decisions tokio made.
That works fine for C#. When writing UI applications all async code started from the UI thread will continue to run single threaded on the UI thread. Everything else runs multi threaded in a thread pool. No need to change any of the async code to work for either.
It's also devilishly hard to understand, I've read a blog [1] on the subject several times and don't always fully grasp the consequences of different options.
[0] https://learn.microsoft.com/en-us/dotnet/api/system.threadin... [1] https://hamidmosalla.com/2018/06/24/what-is-synchronizationc...
https://github.com/davidfowl/AspNetCoreDiagnosticScenarios/b...
https://twitter.com/davidfowl/status/1532880744732758018?lan...
https://github.com/davidwrighton/runtimelab/blob/report/docs...
The green threads research (which are those github issues I already linked to) concluded that it's not worth adding it to NET, because basically now it is too late to retrofit them into .NET, without having the issue of yet another way to color code, for little gain overall.
I also suggest watching the BUILD 2023 ASP.NET panel on the matter.
Not my conclusion at all. I am saying that tooling and base libraries have matured enough to prevent those issues from happening.
> because basically now it is too late to retrofit them into .NET
Nope, read the conclusion. There are other factors not just backwards compatibility (which isn't even a blocker).
Big if, not everyone is using .NET latest on VS, with latest version of every library using async/await.
> Nope, read the conclusion. There are other factors not just backwards compatibility.
Again, I wrote the Github issues regarding green threads on my comment, so I don't know, maybe I actually already read them?
The analyzers are part of the build, they work everywhere (command line, VS code). Either way, I am glad we finally agree that async await "works fine" for modern .NET.
> so I don't know, maybe I actually already read them?
I did, I even responded to your links with the official conclusion from the NET team on those issues. Now maybe your turn to read?
Fixing this would either be a serious break in the ecosystem, or you'd have a new capacity with way too many asterisks to be useful.
The .NET world already spent the budget for big migrations with .NET Framework -> Core (just like Java did with Java 8->9, or Python with 2->3) - as much as we might like green threads, they aren't useful enough (compared to async/await) to justify another break.
As there is nothing being shipped in the box, both suffer from "go hunting" for runtimes, and the interoperability between them.
On Windows, there are bonus complexity points, as they also get to interoperate with COM appartments, and OS async APIs.
If there was a common standard, how do you resolve the additional need of sync/send+static for multithreaded executors?
Agreed, it feels like we're in a worst-of-both-worlds situation. On one hand, tokio is relied upon by thousands of crates, and is very opinionated, meaning it's hard to innovate in the async space. On the other hand, tokio isn't a real standard, so we still get ecosystem fragmentation.
The way async is implemented in rust is actually technically quite impressive, and would almost certainly not exist if there were some official green thread solution.
You could solve async/non-async polymorphism via the introduction of HKTs (and monads) - perhaps eventually they will be forced to do that.
In the mean time, if they can make a few changes like stabilizing TAITs and async traits, that would go a long way to improving the ergos of async.
Rust has stabilized GATs, which are comparable in power to HKTs while having better interop with the language's broader feature set.
Similar to the async runtime most software needs one and most developers don't care much which one they use and are happy with the default allocator. Another similarity is that both are not just some ordinary old library but required by language features. We also usually don't use multiple ones in a single application.
Still we allow the developer to choose an allocator or bring their own one.
For single-threaded vs multi-threaded obviously you'll need to split the APIs, but why can't all 1/N-threaded runtimes work with all 1/N-threaded coroutines (and perhaps another split for no_std)? Ignoring historical differences of course - backwards compatibility means early ones probably can't ever work together. But the ecosystem isn't forever bound to those early implementations.
The solution would probably just to create standard interfaces in std for this so you could just do `std::async::sleep(Duration::from_millis(100)).await` and just delegate the implementation details to the runtime (or something like that).
Like, if I start a thread that calls `sleep(100); timer.resolve(); runtimeInstance.wake()` how is that related to the implementation details?
It's more that a shared API keeps getting presented as an impossibility without stdlib support, and I don't see why that would be true. Stdlib isn't special like that, nor should it be, and nothing seems to be asking for compiler magic (necessary for await in the first place, but not really beyond that).
If anything, the failure of the ecosystem to settle on a shared API seems to imply there should not be a stdlib version - let the competition continue, don't choose any until it's clearly the best choice forever.
The first piece is the compiler support for the keywords, to do the state machine transformation for you. This can only be done by the compiler, so it's agnostic of the runtime. This support also requires some ancillary standard library support for what an async task is, to know what the result of the state machine transformation looks like. In pseudocode, this is:
do_async_task(wake) -> Poll<result>:
if !ready_to_read:
schedule a call to wake when ready to read
return Poll::not_ready
return compute(read())
The third piece of the puzzle is the "schedule a call" part, essentially this is the code that works with low-level system calls like epoll or io_uring or IOCP or boring old select. Except, as you immediately see if you have experience with such system calls, writing that code requires that the top-level event loop essentially be consistently triggering the call to check for new work. So this piece of puzzle, especially for I/O, generally needs to be intimately connected to the top-level executor to work well.
What could the standard library do (or have done) to make things work better? The obvious thing is standardize basic async concepts like AsyncRead or async variant of iterators, except the difficulty with that for I/O in particular is that the library would be standardizing interfaces without providing implementations. Less obvious is baking in a standard I/O event loop interface--a standard way that would allow adding new events to the runtime's main epoll or whatever interface. However, it sort of turns out that many OSes don't actually provide nice interfaces for "wait for I/O or timer or child process status change or GUI event or ..." which is what you really want to have.
I can definitely see why an enforced-shared tightly-integrated epoll-er has implicit performance benefits, but that hardly seems necessary either. And stuff chasing the last bits of performance basically always give up some interoperability.
edit: is it just future/await and nothing else?
https://github.com/smol-rs/smol looks promising simply for being minimal
https://github.com/bytedance/monoio looks potentially easier to work with than tokio
https://github.com/DataDog/glommio is built around linux io_uring and seems somewhat promising for performance reasons.
I haven't played with any of these yet, because Tokio is unfortunately the path of least resistance. And a bit viral in how it's infected things.
But I'm planning on giving glommio at least a whirl.
trait Future {
type Output;
fn poll(&mut self, wake: fn()) -> Poll<Self::Output>;
}
enum Poll<T> {
Ready(T),
Pending,
}
You poll() until it returns Ready.
wake() will notify you when it's ready to be polled again.
That's it.
Tokio and smol and these runtimes only exist to keep track of these, implement their own API, launch some threads, and run this event loop.
I feel like I do this in C++ right now without issue? I routinely mix/match coroutines from multiple runtimes, including one I built myself (which theoretically might could be multithreaded but very much right now is not and I know I rely on that still), one from cppcoro (which is a bit broken--I filed a bug with a detailed analysis, but it was never fixed--so I can only use a few parts that happen to add a lot of value), and one from boost asio (which is very much multithreaded and was a somewhat-impressive retrofit onto a more abstract design purely involving callbacks); I also effectively have a fourth, as another library I am using--libwebrtc--maintains its own I/O thread paradigm, and I have chosen to reinterpret a number of its delegation callbacks into coroutines. It involved some trivial adapters in a few places, but I developed those years ago and have long since forgotten as it all works so easily to willy-nilly co_await anything I want from wherever I am... is this really so difficult?
That said, if you don’t use shared state that allows multiple borrows, you won’t see state changing between futures even in the single threaded cases due to the ownership model of rust.
With a multithreaded system, async or not, you have to worry about the concurrency issues that come up when sharing data between multiple threads, because that’s what you’re doing.
It’s odd how Rust ended up with the worst of both worlds by default. I think people got overconfident because Rust otherwise handles multithreading so well.
Except that async code written this way in JavaScript/Typescript often ends up being subtly broken by evolution in where the awaits occur as the software is maintained. IMO, it’s generally better to design async code with a shared-nothing mentality anyways.
I think there should be no default for how to call an async function from another async function. Both waiting for a response and not waiting (starting a "background task") should be acknowledged in the code. Perhaps not allowing a promise return value to be silently dropped would be enough.
Sync functions are easy in comparison.
Edit: I guess there is a lint rule:
Like, here: if I am in my single-threaded runtime and I await something on a different runtime with a billion threads, MY continuation does NOT need to be able to resume on any of those threads as it CAN'T. To achieve "seamless interoperability" I just need to be able to await the other routine and resume when it completes, not somehow make the two runtimes merge into one unified one and violate their constraints. The ONLY data from my coroutines which should end up on a different thread is what I explicitly pass to the routine, not my continuation.
There are some pretty common reasons why a future in not Send:
1. It is reliant on some thread-local state in which case you can't move it to another thread 2. It uses something which relies of being single threaded for sounds. An example would be `Rc` the standard reference counted pointer in the std. It uses a `usize` for the refcount so it is not safe that have two `Rc` for the same data on different threads. If you need a reference counted pointer that is thread safe you need to use `Arc` which uses an `AtomicUsize` for the ref count and so is Send.
> I just need to be able to await the other routine and resume when it completes, not somehow make the two runtimes merge into one unified one and violate their constraints. The ONLY data from my coroutines which should end up on a different thread is what I explicitly pass to the routine, not my continuation.
Sure, and you could do this in Rust now perfectly fine. Spawn a future on a separate runtime (or a CPU intensive task on a regular thread) and await the result on the current runtime. But by default what happens whenever you hit an `await` is that the coroutine is suspended and goes onto the runtime's run queue until it is woken back up and gets rescheduled. In Tokio's multi-threaded runtime it can be rescheduled on next wake on any worker thread so it must be `Send`. If you use the single threaded tokio runtime there is only one thread so it doesn't need to be `Send`. And even in the multi-threaded tokio runtime you can still spawn tasks that are pinned to the current worker thread using LocalSet.
In writing application code this is (to me at least) mostly a non-issue. Most futures will be Send anyway so the Send bound is not a big deal. But if you do have something that is not Send then you can always use LocalSet to spawn it. The issue I think is really in writing library code where you start to have to add Send bounds everywhere so it jives with multi-threaded runtimes. Like say you have a trait with a method that returns a `Stream` but the concrete type of the `Stream` is not important as long as it produces the required output. So you have
``` trait MakeThingStream { fn make_it(&self) -> Box<dyn Stream<Item = Thing>>; } ```
Well now all the compiler knows is that the output implements `Stream<Item = Thing>`. But this may not be Send so you'll get compiler errors if you try to use this in a multi-threaded runtime. So you add Send/Sync bounds:
``` trait MakeThingStream { fn make_it(&self) -> Box<dyn Stream<Item = Thing> + Send + Sync>; } ```
Great, now it plays nicely with multi-threaded runtimes but even if it's being used in a single-threaded runtime you still require the Send/Sync bounds.
It is absolutely possible to make a runtime agnostic library that can work over multiple runtimes. With the trust-dns libraries, we’ve managed to provide a resolver which is capable of working on async-std, Tokio (default), and even Fuchsia. It’s harder and takes planning, also to be fair and fully transparent we haven’t achieved this for all features, like DNS-over-quic.
> We should have an official runtime, officially managed, and guided by the same thoughts that guide the rest of the language.
I disagree. Rust is a systems level language capable of being used to build Operating Systems or other embedded tools, having a single runtime would make async Rust something you could not use in that context.
“amateurs talk strategy and professionals talk logistics”
Rust community is endlessly talking and obsessing with strategy where as average Rust user suffer from lack of logistics concerns about libraries / runtime usage etc.
I agree that in some areas there could be better guidance. Is Tokio the runtime most people choose? Yes. Would most people be fine choosing that for their daily work? Yes. Might you want to choose a different one? It depends on what you’re doing, others have different goals and tradeoffs. Are there interface choices regarding things like Send + Sync or IO interfaces/traits you pick that will have impacts on how you structure your code to make it portable across runtimes? Absolutely.
And finally, can Rust be better in regards to async development? Yes, everyone agrees that it should be. My big thing is that we really need async traits in the language. We have an excellent work around with the async-trait macro until we get support for it in the language, but you need to discover that, and then recognize some of its idiosyncrasies in certain situations.
But I guess we can go like this:
1) Will community welcome a sync crate ecosystem? Yes.
2) Should people write sync code at all? depends...
3) Can some one write RFC for rust team if they need some feature in Rust? Certainly.
4) Should someone write libraries missing in ecosystem? Yes, community will love it.
Now everything is well and good.
I’ve regretted that every time I’ve done it in my career, especially in network programming. All the different error conditions and potential blocking conditions that tcp connections can end up in are just easier to deal with on async interfaces.
I guess a different question I would ask is, what can we do to make async programming easy enough in Rust such that people don’t feel a need to reach for synchronous/blocking interfaces?
I stopped really paying attention to Rust about 5 years ago, and am asking purely out of ignorance/curiosity, but has the community/leadership approach changed much since then? I remember async being the Shiny New Future that was talked about a lot back then, but it certainly seems like what's been added has not really done well?
My (probably incorrect) understanding is that "async" arose from Javascript. It arose because pure event driven code is error prone and hard to get right compared to linear (stack based) code. The usual solution is threads, be they real or lightweight (aka cooperative multitasking) - but Javascript doesn't have threads and never will. Compared to pure event driven code using zillions of objects to save state, the pseudo stack based async solution is indeed a blessing.
async is in effect a poor mans emulation of lightweight threads. It comes at the cost of needing language syntax to support it ("async" and "await") and it creates different colours of code, ie code that can't be mixed. The end result is parallel implementations of lots of libraries, leading to the situation the article and above comment both moan about.
Lightweight threads / green threads achieve the same outcome as async, but without the downsides. No language extensions, no coloured code, all existing API's remain backward compatible. Javascript didn't have a choice, but why any language that does have a choice would use the async solution has me completely baffled. It's not like we didn't have numerous examples such as Elixir or Go, yet Rust went with async anyway.
I gave a talk (with transcript) about this here: https://www.infoq.com/presentations/rust-2019/
> Lightweight threads / green threads achieve the same outcome as async, but without the downsides.
This is not the case. Both have pros and cons.
Thanks. A fascinating history. Do keep making them.
One thing that had me scratching my head was the "green threads made C calls slower" comment. I don't understand why C would care where you call it from.
> This is not the case. Both have pros and cons.
Your talk highlighted on green thread con I hadn't though of. It hadn't occurred to me green threads introduced coloured code, just like async does. That was made plain by it needing a std::io implementation.
But that isn't an additional con green threads have over async - it's a con they both share. While I get that green threads didn't interact with native threads very well, but I'm making a bet that was because they tried to hide the colours (different IO library) it needed, so the programmer didn't have to care. Async would have had the same problem had it tried to hid the colouring it introduces, but they solved that by not hiding it.
Async warts over green threads of introducing a new syntax and a slightly different programming style remain.
> I don't understand why C would care where you call it from.
The details here differ based on what kind of green threads you are implementing, but the core of it is, they're cheaper than regular threads because they do not use a normal stack. C expects a normal stack. Bridging this gap has a cost. You also have to manage the interaction between the GC and C, which can have a cost. If you're curious about specifics, one example of this is cgo: https://go.dev/src/runtime/cgocall.go Go has changed strategies here several times throughout its history (as did Rust when Rust had green threads), so you may find other information that's older as well.
I should reveal at this point I've created protected mode x86 OS's from scratch, written BIOS's and what not, all done in C, so I do know a bit about C and stacks.
As I expected there is nothing in cgo that suggests C that cares about a stack. That's not surprising as with the exception of esoteric things like setjmp, and backtraces, C doesn't care. You can happily malloc a block of memory and point the SP there, push the args and call a C function, and it will do it's thing and return. It's vaguely possible the OS may get pissed off that the stack isn't where it thought it should be - but the user space C function won't notice.
What cgocall() (the function that handles go call's to C) spends most of it's time doing is tell the green thread scheduler what is happening. I'm guessing the reason for that is the C code is effectively code of a different colour - ie it's code that could be using blocking I/O calls. If the C function does block it won't stop just the green thread calling it, it will block all of them. I imagine is not considered acceptable in Go. A work around would be to move the green thread to a different native thread while the C function is running. Maybe that's what all that bookkeeping accomplishes does. As you say, and as I can see in cgocall(), the overhead of bookkeeping involved is literally orders of magnitude bigger than the overhead of the C call itself.
And as you also say, that overhead isn't acceptable for Rust. The solution Rust has implemented for async is effectively ignore the problem, so if a async function calls a C method and that C method blocks, then every async task stops until that C function returns. It would have been a perfectly acceptable solution for green threads too. But I'm guessing the original Rust green thread went for the Go "make the library hide the problem from the programmer" approach, and found itself stuck with a whole pile of overheads that ended up being unacceptable for a systems programming language.
If so, the solution wasn't to throw out green threads and adopt the async solution. That was akin to throwing the baby out with the bath water. The simple solution was to just take the async approach and make the issue of blocking C calls the programmers problem, as opposed to hiding it with the runtime libraries.
If they had have gone that route even handing blocking C calls could have been made relatively straight forward - just provide a library function calls the function it's passed in it's own thread. (Maybe async already provides a similar function now?) Effectively that lets the programmer choose when to take the C call overhead Go imposes on every call, and when to avoid it.
Right now, it looks to me like my opening comment still stands - green threads (although not Rust's initial implementation) would have been a much better solution over async to the multi tasking problem. At the 1000ft view, green threads and async are very similar. Both get their speed by using event driven I/O rather than blocking I/O, and thus avoid the overheads of OS task switching. The key difference is where green threads store state on a separate stack (a technique so wonderfully efficient we use it everywhere), async stores it in manually allocated block that must then have data copied into it, and later freed. That manually allocated block creates a lot of overheads, both in code and at runtime, that green threads don't have.
Okay well again, I'm trying to be very broad and vague here, because the details do actually matter but differ between systems. C in a general sense doesn't care, as you elaborate, sure, but because these stacks are so small, and C code doesn't know how to expand the stack (since there's no API to do so), you run the risk of overflowing the stack. So in practice, that stack usage does matter, and the way that you protect against this is to set up a regular sized stack, swap to it, and make the call. At least, in this specific implementation. http://manticore.cs.uchicago.edu/papers/pldi20-stacks-n-cont... talks about tradeoffs of six different ways of implementing this kind of thing, for example. (both Go and Rust tried the "segmented" strategy here and threw it out, for example.)
> (Maybe async already provides a similar function now?)
Many implementations provide a threadpool for you to throw blocking stuff onto, yes. That's up to the given runtime. But again, that's purely for the blocking semantics, it isn't about calling into C vs calling into Rust.
Anyway if you truly want to understand this space I would encourage you to continue looking into it, but when it comes to demonstrated performance in the real world, the green thread strategy loses out. There are other great reasons to choose that model, but for Rust's systems language goals, as well as its performance goals, async/await is the only design that's made sense.
Odd they didn't compare the most common strategy used in practice, which is the one the linux kernel uses. The technique is described in mmap(2), under the MAP_GROWSDOWN flag. Even if you allow for a 64Kb stack for each green thread a 32bit machine has enough virtual address space for thousands of stacks. If you need more add an option to trim down the stack size.
> But again, that's purely for the blocking semantics, it isn't about calling into C vs calling into Rust.
Yes, it's blocking semantics. But the reason given for abandoning green threads was those calls from Rust to C were too slow in green threads, and the only reason I can see that would be is the library is attempting to hide those blocking semantics by intercepting every C call. It it didn't there would be no speed disadvantage.
Yes, intercepting slows down the call by an order of magnitude. But there is another solution - don't intercept the calls, let the programmer handle it instead. That's the solution async adopts. If you are going to claim green threads are slower than async then it's only fair to compare apples with apples, and that means comparing implementations that do it the same way.
Mind you, it's purely a guess on my part that the old green threads implementation slowed C calls by intercepting them, so it's purely a guess we aren't comparing apples with apples. The guess is based on the fact there is no other reason green threads C calls should be slower, as C doesn't care one way or the other.
> There are other great reasons to choose that model, but for Rust's systems language goals, as well as its performance goals
I can't see what systems language goals would be broken by green threads - but then I'm not familiar with them. Apart from the C call thing, green threads should be faster as they are storing data on the stack rather than copying it into a manually allocated block. Since the C call thing is looks to be a problem with the design choices of that early Rust green thread model, I don't trust the claim an implementation of green threads that makes the same tradeoffs as async currently does would be slower. And green threads does provide a much cleaner API.
But I guess the response to my whinging at this point is "patches are welcome", or rather an appropriate green thread implementation.
And it's included in a set of independent benchmarks of http servers written in variety of languages: https://www.techempower.com/benchmarks/#section=data-r21&tes... May (and Rust) put in a very good showing there, may-minihttp taking out 2nd spot. Another Rust library, xitca-web, takes out 3rd spot. Neither may-minihttp nor xitca-web use async, but there are other Rust async implementations that come close to them. I'd call it a wash.
From that I'd say may's green thread implementation is on a par with async speed wise.
Errrk. I was looking at using it (because async really does suck from a usability point of view compared to green threads). Do you have a link?
Hmmm. Is it TLS consuming too much stack? https://github.com/rust-lang/rust/issues/111272
That would be an issue for green threads. And other things, as I discovered when I took a brief look at the may code to see if they handled stack allocation. Turns out may doesn't don't handle it directly - the standard library (nightly) has a way of creating stacks for co-routines (generator::Gn). May's green threads are just co-routines, and the Rust nighly library provides the stack.
That means if it is the issue I linked to, it's a bit unfair to blame it on may. The same bug will manifest itself any Rust nightly generator that calls TLS.
Probing further, it generator::Gn creates using stack::Stack, and stack::Stack allocates stacks using malloc. And yes, that guarantees stack overflow will cause UB of the worst sort because it just overwrite the next malloced block. Someone should lookup "man 5 mmap" on Linux and BSD. Both have ways that create stacks behave very nicely, including causing a hard fail if they overflow rather than UB. I presume Windows has a similar function.
To repeat the point I keep making: all these issues with green threads aren't intrinsic issues to the concept. They arise because the initial Rust implementation wasn't well designed, and not implemented particularly well either.
> The key benefit of green threads is that it makes function colors disappear and simplifies the progr'samming model.
As a point of order, no, green threads don't make colours disappear. They can't as the whole point is to run multiple tasks, so no green task can be allowed to make a blocking I/O call like native code does, so you have re-do every I/O library using non-blocking I/O. And thus green threads must use the non-blocking version of the library, aka as a different coloured code.
Where green threads are different to async is the language library can make the colouring disappear for green threads. It does that by, on every I/O call, checking if a green thread is making the call and switch between blocking and non-blocking I/O accordingly. That incurs a speed penalty of course. And it doesn't just hit green thread code, it slows down native threads too.
Looks like .net decided that overhead is too high to bear. Fair enough - but that's a consequence the decision to hide coloured code, not green threads per se.
While you could do the same trick to hide blocking vs non-blocking for async code too of course, it wouldn't hide colouring. That's because async colours code in other ways too - for example it introduces a whole now call / return syntax. Unlike "not needing colours", not needing a new syntax is a real advantage of green threads over async. Another one is saving state on the stack rather than a malloced block. (If writing function locals to a malloc'ed block was faster than pushing them on a stack was faster we would do it everywhere.)
Hmmm, I wouldn't put it that way. There's Tokio as the defacto default runtime with a large ecosystem for "standard" usecases. Axum (Hyper, Tower middlewares) or Actix, SQLx/Diesel/Rust-Postgres, Request ... are a wonderful and for my limited usecases rich ecosystem.
Right now cross-runtime libraries are mostly written special-cased: one feature flag for tokio, another for async-std, maybe one for smol if they feel fancy. Almost none for glommio or other runtimes. That introduces a huge burden for libraries, that would rather depend on a single API.
Rust shouldn't have an official runtime; it should have APIs that make possible to write libraries that don't dictate which runtime you must pick.
Instead, Rust is waiting for patterns to emerge in the ecosystem, so that the universal API is compatible with all of them. It's a much safer route, but it also means waiting for the ecosystem to fragment. This is where we are now, but it's necessary and will get better over time.
Edit: async readers/writers are a good example, because it seemed trivial to set in stone, except now we have io_uring that might require the API to move to the kernel instead of just holding &mut.
I didn't agree with this sentiment though:
> In a recent benchmark, async Rust was 2x faster than threads, but the absolute difference was only 10ms per request. To put this into perspective, this about as long as PHP takes to start. In other words, the difference is negligible for most applications.
This statement is an ugly thorn that sticks out of the otherwise well written and reasoned article. It hurts me deep on the inside when I read stuff like this.
That's a lot.
However, CPUs can also execute more than one instruction in a cycle so effectively, you can have more instructions than cycles.
If you’re in single-digit mikes at the tail you’re not fucking around with someone’s green threads, and at 100ns you’re in an FPGA or even ASIC.
To serve a web page? 10ms, eh, I’d rather not spill on purpose but it’s a very, very rare professional CS:GO player who can tell. If my code is simpler and cheaper to maintain and more fun? Maybe I pay it.
What I don’t want is to ‘static bound shit to burn millis. Burning millis should buy me a beautiful sunset and a drink with an umbrella in it.
Latency is not directly related to throughput.
This phrase alone is utterly incoherent.
Interestingly, when working with a limited number of threads, the thread approach is actually faster in that benchmark. So in practical applications, the differences are marginal and likely lean towards threads.
But even if this weren't the case, context matters. A 10ms discrepancy in a web request might be acceptable. However, in a high-performance networking application - which, let's be honest, isn't a common project for most - it could be significant.
The native model falls apart under C10M, but to be fair so does traditional epoll/queue/iocp dispatching coroutines model of solving C10K. That's where you start having to keep the network stack and application data plane colocated in the same context as much as possible. That can be done with something like DPDK to keep those both in user space, or Netflix is known for their FreeBSD work making KTLS and sendfile kiss in order to keep the data plane completely in the kernel.
async runtimes can just perform better at scale or reduce the amount of resources at scale. Where "at scale" means a concurrency level of >= 10k in the last benchmarks I did on this.
About that. Modern PHP in event loop with Swoole have response times of <1ms.
So 10ms is some legacy Apache/nginx mod_php/php-fpm number.
And even traditional php-fpm+Linux response times is around 2-5ms depending on configuration.
The Rust and tokio folks are working on difficult and complex problems, I appreciate and thank them for the work they're doing to improve server and desktop app performance everywhere. We all have multicore machines and it would be great if we could use more than 1/8 or 1/12 (or whatever high thread count of your beefy servers) of our hardware.
The Rust multithreaded thread memory management (and Arc and so on)* causes me to be uncomfortable because of a key lesson I've learned is that you cannot scale a program by adding threads and expect it to accelerate mutate access to the SAME memory location. Single threaded memory mutation performance is a fixed known quantity. Adding threads with contention for same memory location causes throughput and latency to be slower to a particular memory location at single threaded speeds because you need mutexes or a lock free algorithm to communicate safely.
To accelerate data fanout or storage (writing to memory from multiple threads) you need a shared nothing architecture or sharding.
This means that when you reach for threads and I'm guessing you're wanting to reach for threads for acceleration and performance you need to design your data structures to not share memory locations. You need to shard your data.
EDIT: I originally said Sync + Send.
I can see that Send/Sync by itself does not tell you if the data is read only or just internally synchronizing mutation.
It is probably important here to realize that async solves concurrency, not parallelism. You can use async with a single threaded runtime for I/O concurrency and mix that with threads for computational parallelism for long running jobs.
That said, there may be some benefit a multi-threaded runtime would have for the typical I/O bound app (after working around lifetime limitations by adding Send/Sync to data structures). This is because I/O bound programs and those requiring computation are not mutually exclusive and there is always some amount of computation going on, so there may still be some benefit. I doubt a synthetic benchmark would answer this as those typically don't measure any actual work performed, but just "requests/sec".
In my experience, it's impossible to mix threads and async tasks. They can't communicate or share state. Threads need locks, while async tasks require an awake mechanism. If you just stick to unbounded channels that don't block on send, you can get far, but in 99% cases you will need to decide upfront on a specific approach.
A pattern that has worked quite well for me is to use
``` struct ComputeTask { some_state: SomeState, completion: Sender<Output>, }
impl ComputeTask { fn run(self) { .. do you compute-intensive stuff
self.sender.send(output);
}async fn do_stuff() -> Result<Output> {
let (tx,rx) = tokio::sync::oneshot::channel(); let task = ComputeTask { .., tx }
rayon::spawn(move || task.run());
rx.await
```
https://calpaterson.com/async-python-is-not-faster.html
I think the idea is that while your blocking waiting for IO in one task you can serve a different task, potentially from a different user. Coroutines, green threads, communicating sequential processes as in Go or Occam.
async can actually _increase_ the number of syscalls your application performs.
>don't you havev to allocate that input buffer each time?
Have to? No, you could pre-allocate and reuse buffers. It is less straight forward than the buffer per thread strategy, but possible.
E.g. having a thread that spends perhaps 1% of the time with state mutation vs 10 threads spending each 3% of the time with state mutation. You have smaller efficiency per thread, but still higher efficiency overal
http://www.frankmcsherry.org/assets/COST.pdf
Which I think is about how single threaded programs are faster than scalable but slow multithreaded systems.
I think you might be right and that's why there is ReadWriteLock or RwLock for single writer taking turns.
If you have a counter or a data structure you're mutating in every request then you'll hit lock contention.
Blindly using Arc<Mutex<T>> without considering access patterns is a software architecture problem, not a problem with Arc or Mutex.
That is a very weird argument to make. Tokio has very convenient APIs (LocalSet + spawn_local) for spawning non Send futures that you temporarily await (which really is the only useful thing non Send futures can do).
If anything tokio significantly improved the user experience of async in Rust in general because it promoted Send futures.
LocalSet + spawn_local are great, and I wish more developers would know about them, but the Tokio tutorial [1] doesn't mention that and focuses on the multi-threaded runtime instead. AFAIK LocalSet is only mentioned in the docs [2]
[1]: https://tokio.rs/tokio/tutorial [2]: https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html
Patches welcome: https://github.com/tokio-rs/website/
"Doing the right thing" can vary based on the context. For instance, in embedded systems where threads aren't available, requiring futures to be Send is unnecessary. Thankfully, the standard library does not enforce this and neither does Tokio with spawn_local, but embassy exists because there's a genuine need for async frameworks tailored to the unique constraints and requirements of embedded systems.
In general Rust has tried hard to improve on the developer experience of C++ by providing more safety in the language and better defaults in the standard library. So it's interesting that both languages have now ended up in a similar same place for async.
(C++20 coroutines finally enable sensible async libraries, but code written against a higher-level library isn't easily portable to another one even though both are using the low-level language coro primitives.)
> "Freely after Stroustup: Inside Rust, there is a smaller, simpler language that is waiting to get out. It is this language that most Rust code should be written in."
Maybe there's a Meta-Stroustrup's Law in effect:
"Every successful language eventually becomes one which contains a smaller, simpler language struggling to get out."
It happened to C++ and Java and JavaScript, now Rust seems to be reaching that point.
Edit:
> Maybe there's a Meta-Stroustrup's Law in effect:
> "Every successful language eventually becomes one which contains a smaller, simpler language struggling to get out."
Corollary: every simple subset of language contains missing features dearly needed by someone else.
This could even be said about larger languages, though. If Rust committed to pleasing everyone, it would have an optional garbage collector, lifetime annotations would be optional, and there would be an interpreted runtime available as an alternative to the compiler. Those are features that some people do dearly need for some tasks. There's a point where you have to stop and put limits on what the language is actually for and what it's not. It seems to me that much of the push to add async/await in the first place was from people who really should have been using Go, Java, or Node, but wanted Rust to be their "everything language." It's okay to say, "This language is for writing performant systems applications. It's not a fullstack language for writing a webapp."
I work a lot more in Rust than I do in go, but I think each language made the trade-off that made most sense for that language.
Wasm has been really great at shipping the MVP, but they are pretty slow about shipping the many features that build on it. In general, this makes sense as the system can't be changed once its stable. But it also means that a lot of things are still in limbo and will probably be for the forseeable future.
> I think each language made the trade-off that made most sense for that language.
Definitely! Go is meant for backend application logic computing where you can provision tons of ram and ignore the issues of gc. Rust targets a larger domain, less application logic in particular but the whole range of system programming. Also including applications but also low level libraries, places without an OS, etc. I think if Rust really wants to be low-low level, then not shipping an async runtime is a must, even if the std crate is present. Providing features for libraries to support multiple runtimes? Sure. But don't apply solutions that (mostly) work for Go to Rust's problem domain.
I concede that development of post-MVP development has been slow, but I am also optimistic for the near future based on recent progress. Are there particular things in limbo that you're particularly interested in or concerned about?
I do think the post is a bit unfair in this sense, it rightly identifies the problems of Send and 'static. However, it also suggests Arc and Mutex are *the* solutions for shared state in async, but suggested channels for the threads example.
The problem of function colouring and Send/'static bounds are the significant hurdles with async Rust, shared state is something that needs to be resolved whether using threads or async.
This is where Go really shines. Goroutines may not be "right" or "good", but they are very intuitive, and maintainable. Performance isn't bad either.
In Rust, there's the May project that is very similar and should really get more attention.
This circle will repeat its self over and over again
The author mentions
> If async is truly indispensable, consider isolating your async code from the rest of your application
I think ALL async code should be generally isolated.
Are there languages that provide foundational priority to asynchronous code yet supports good intermingling of sync and async in the same codebase? I maybe missing the point about isolation, but the mix of sync and async code gets bad really quick.
Rust, on the other hand, took a different route. Green threads don't integrate smoothly with code interfacing through FFI. Moreover, Rust's async model doesn't require a garbage collector.
I understand that making async HTTP request is a fundamental concept. However, I question why we should recommend a more complex solution when there are simpler alternatives that still leverage Rust's capabilities.
I can't help but wonder if there are two sets of interfaces necessary? a set of standard single-threaded traits and a set of multi-threaded traits? would that be sufficient?
As an aside, what workloads require true multithreaded reactors as opposed to a runtime which uses multiple singlethreaded reactors?
For example, DataFusion spreads query processing over multiple cores by having the data flow be a "streaming DAG of record batches" (or something like that), as in futures::Stream.
https://docs.rs/datafusion/latest/datafusion/
https://docs.rs/datafusion/latest/datafusion/execution/trait...
Also, impl trait projections (ability to use Self::Foo associated types in async functions in traits): https://github.com/rust-lang/rust/pull/115659
So, obviously having to sprinkle Arc and Mutex all over the place sucks as a developer experience. But how much does that really impose in terms of runtime overhead? Both of those structures tend to perform decently in the single-threaded case. It's obviously useless work, but I'd be surprised if that shows up on any profiles as a bottleneck.
Why do you think Java recently went from N:M scheduling? Surely there's something to it?
https://lobste.rs/s/iovz9o/state_async_rust
The tl;dr is that I think this entire async concern stuff is ridiculously overblown. I suspect the vast majority of Rust devs, like myself, not only accept the current state as "fine" (couple things to work on) but are very happy with the decisions tokio made.
Things like "oh no you have to use Arc" are really acting like that's more than a trivial change. Or "you have to use Mutex" when you don't. Or "you can accidentally block" as if that's not the case with regular threads too, and in that case I acknowledge that async can make things trickier there. Or "tokio is so popular" as if that's not exactly because of the decisions it made early on that appealed to rust developers.
Sorry but it's just not that big of a deal. The warts I run into are in cases where I'm trying to do stuff like zero copy, async, abstracted deserialization. That can be a pain right now (and is being worked on). 99% of the time it's a matter of just writing `async` or `await` and not worrying about anything. In fact, I almost never use `tokio::spawn` anyways except at the binary level - these problems virtually do not impact me.
Source: I have written 10s of thousands of lines of async Rust, probably 100s of thousands.
I'm confused, what are you suggesting? That Rust hits a global sweet spot already, the complainers are struggling because they are holding it wrong, and there shouldn't be an attempt to change anything?
As for changing things, I wouldn't really change any issues brought up here. I'd like to see some things smoothed out, like async traits, tooling to identify hot loops that are blocking without yielding, and things like that. Otherwise, nope, working as intended as far as I'm concerned.
Also this allows a wide range of topics - related words - to work as wordplay.