Local async executors and why they should be the default

The whole thing just reads... ugly and inconsistent. It needs too much already-accumulated knowledge. As the article correctly points out, you need a bunch of stuff that are seemingly unrelated (and syntactically speaking you would never guess they belong together). And as other commenters pointed out, you need to scan a lot of docs -- many useful Tokio tools are not just not promoted, they are outright difficult to find at all.

Now don't get me wrong, I worked on projects where a single Rust k8s node was ingesting 150k events per second. I have seen and believed and I want to use Rust more. But the async story needs the team's undivided attention for a long time at this point, I feel.

Against my own philosophy and values I find myself attracted to Golang. It has a ton of ugly gotchas and many things are opaque... and I still find that more attractive than Rust. :(

This article is a sad reminder for me -- I am kind of drifting away from Rust. I might return and laugh at myself for this comment several months down the line... but at the moment it seems that my brain prefers stuff that's quicker to grok and experiment with. Not to mention writing demos and prototypes is objectively faster.

If I had executive power in the Rust leadership I'd definitely task them with taking a good hard look at the current state of async and start making backwards-incompatible changes (backed by new major semver versions of course). Much more macros or simply better-reading APIs might be a very good start. Start making and promoting higher-order concurrency and parallelism patterns i.e. the `scoped_pool` thingy for example.

a) Saying Node + Deno are good is a stretch. Node has horrible performance, even for simple routing. And I'll source that[0].

b) Saying that adding `Send + Sync + 'static` bounds is a serious burden is, to me, overstating things.

> the far better model for writing performant servers.

It's completely workload dependent. For a chat server it's almost definitely not going to be more performant and you may end up with worse latency.

> it only costs you friction everywhere else in your entire codebase, and quite often performance as well.

I am unconvinced tbh. I do not believe that adding Send + Sync + 'static bounds is onerous, I do not believe satisfying those bounds is hard (it's almost always just a matter of moving the value), and I do not believe that the vast majority of programs benefit from TPC architecture.

I recognize that there is a problem here - that we are optimizing for one runtime at the expense of others - but I am not convinced at this point that the problem matters.

[0] https://www.techempower.com/benchmarks/#section=data-r21&tes...

Don't do that.

Edit: Thanks to the mods or whoever fixed it.

The only reason all this async stuff even exists is because we want concurrency. We want to say "while this one task waits for I/O, this other task will do stuff". So it's not too surprising to me that an intro to async would include synchronization primitives. Those primitives aren't really "thread"-specific if by thread you mean OS thread. When you do async like this, you're basically re-implementing OS threads in user space.

I cannot agree more with that. As someone who's done a good deal of Java in my day job, I can tell you a thing or two about spawning threads willy-nilly. At least it is easier to avoid in Rust, but I'd still prefer it the other way round: opt-in, instead of opt-out.

Why so? Libraries like quinn[1] define "no IO" crate to define runtime-agnostic protocol implementation. In this way we won't suffer by forcing ourselves using synchronization primitives.

Also, IMO it's relatively easy to use Send-bounded future in non-Send(i.o.w. single-threaded) runtime environment, but it's almost impossible to do opposite. Ecosystem users can freely use single threaded async runtime, but ecosystem providers should not. If you want every users to only use single threaded runtime, it's a major loss for the Rust ecosystem.

Typechecked Send/Sync bounds are one of the holy grails that Rust provides. Albeit it's overkill to use multithreaded async runtimes for most users, we should not abandon them because it opens an opportunity for high-end users who might seek Rust for their high-performance backends.

[1]: https://github.com/quinn-rs/quinn

A Send bound is no great burden. The only type that you will regularly interact with that is not Send is the lock type from a Mutex or RwLock, which is good because if you hold it across a long await you can slow down your app, a bug prevention mechanism that does not exist in straight multithreading. The only point that actually illustrates a threading-caused problem is the thing about multithreading sockets, which it admits is almost imperceptible, and which you can also solve by not doing that.

Almost everything the author identifies as a parallelism problem is a 'static problem. Tokio is missing a scoped spawn like std has, and if it gained one then the much described multithreading woes would reduce to basically nothing.

Actual title: Local Async Executors and Why They Should be the Default

Posted title: Async rust – are we doing it all wrong?

Really? Why this kind of terrible editorializing?

So what I'm writing uses threads. About 20 of them. They're doing different things at different priorities towards a coordinated goal. This is different from the two usual use cases - multiple servers running in the same address space, and parallel computation of array-type data.

Concurrency problems so far:

- Single-thread async is simple. Multi-thread async is complicated. Multi-thread async plus other threads not managed by the async system isn't used enough to be well supported.

- Rust is good at preventing race conditions, but it doesn't yet have a static deadlock analyzer. It needs one.

- Different threads at different priorities do work in both Linux and Windows, but not all that well. With enough low-priority compute-bound threads to keep all CPUs busy, high-priority threads do not get serviced in a timely manner. I was spoiled by previous work on QNX, which, being a true real-time operating system, takes thread priorities very seriously. On QNX, compute-bound background work has almost no effect on the high-priority stuff. Linux just doesn't work well at 100% CPU utilization. Unblocking a lock does not wake up a higher priority waiting thread immediately. This can delay high-priority threads unnecessarily.

- The WGPU crowd has spent a year getting their locking sorted out so that you can load content into GPU memory while the GPU is rendering something else. It's a standard feature of Vulkan graphics that you can do this, but it has to be supported at all levels above Vulkan too. For me, that's WGPU and Rend3. That stack is close enough to ready to test, but not ready for prime time yet.

- There's no way to cancel a pending HTTP request made with "ureq". "reqwest" supports that, but you have to bring in all the async and Tokio stuff, which means you now have multi-thread async plus other threads. This is only a problem for what I'm doing when the user closes the window, and the program needs to exit quickly and cleanly. I'm getting a 5-10 second stall at exit because of this.

- Crossbeam-channel is not "fair"; it's possible to starve out some requests. Parking-lot is fair, but doesn't have thread poisoning, which means that clean shutdowns after a panic are hard.

- Running under Wine with 100% CPU utilization with multiple threads results in futex congestion in Wine's memory allocation library, and performance drops by over 99%, with all CPUs stuck in spinlocks. The program is still running correctly, but at about 0.5 frames per second instead of 60 FPS. Bug reported and recognized by the Wine crew, but it's hard to fix. I can make this happen under gdb running my own code and see all those threads in the spinlocks, so I was able to file a good bug report. But I haven't generated a simple test case. It's a Wine-only problem; doesn't affect Microsoft Windows.

So that's life in a heavily threaded world.

Individual CPUs have not become much faster in over a decade. Everybody has been stuck at 3-4 GHz for a long time now. CPUs with many cores are widely available in everything from phones to game consoles. To use modern hardware effectively, you need threading.

One of life's big questions.

Rust software should use async tactically - in places where it is needed. Unfortunately handling http, which is a large part of many applications is actually a place where async has benefits. But if you plan to run your http behind nginx anyway (for TLS termination) even there using blocking http server might be a good idea.

> If you write regular synchronous Rust code, unless you have a really good reason, you don't just start with a thread-pool. You write single-threaded code until you find a place where threads can help you, and then you parallelize it,

I disagree with this one. When you work on a software project you should have the basic architecture figured out already, and main part of that is breaking your software into structurally parallel parts that can work independently. Adding ad-hoc parallelism after the fact works only for small scale things and will lead to rather accidental concurrency architecture.

Then for each part (groups of threads), figure out if it *needs* async. Between each part you'd communicate via channels or some shared data structures that rather easily can be made to work with both async/blocking code.

So e.g. an async http server, benefits from lightweight async concurrency, makes rpc-like channel-based calls to blocking IO / CPU/bussiness-logic intense workers (that don't benefit from async) where it makes sense. Each part is written in the best "type of Rust for its use-case".

Or if you need ability to cancel certain computations inside the larger framework (e.g. simulating agents etc.) you might want to nest async executor inside a blocking code.

Note: there's a lot of types of program archetypes out there (CRUD, ETL, data-intensive, embedded, frontent SPA, native mobile app) and I've noticed that many people are boxed in the type they happen to work on. CRUD applications (which are very common) are often 90% http handling-based and it might make sense to write them whole in async Rust.

Re-read this a few times, and I'm fairly convinced it is not generally true. The author is also being a bit confusing about what exactly he means by "socket" here. Because while it's true that multi-threading over a server socket (e.g. the one that binds to the port when you launch the server) will not yield performance gains, multi-threading clients (that have their own sockets, including file descriptors) definitely will. That's the whole point of nginx thread pools[1]. Note that nginx does zero "CPU-bound" work, it literally just serves files.

Node/Deno being single-threaded is purely a limitation of Javascript. Tomcat, Jetty, etc. are all multi-threaded. I'm a bit tired, so I can't comment on the rest of the post in detail, but this was a bit of a red flag.

[1] https://gist.github.com/denji/8359866

If rust would only be targeting web server programming the right decision might have been no async and green threads, it's just much easier to use.

But that rust would likely never have succeeded as most of it's initial success cases where for use-cases where you wouldn't want green threads.

Nicely we might still get no async and green threads: In form of run times which run WASM compiled rust code in a node like fashion. Probably in combination with some serverless/edge-compute providers which hopefully will be nice to use.

The thing I enjoy about async in Python is that it's very easy to write "thread"-safe code - you know exactly where you might give up execution context, and 90% of the time you have no need for mutexes and locks. But as this article complains, in Rust, it seems to be sync or multi-threaded.

Part of the problem is that, in Rust, stack memory is easier to use than heap memory. When writing traditional threaded code, this isn't much of an issue because naturally most of our code is working with values on the stack.

BUT: When we look more closely in how async works in C# or Javascript, the compiler, under the hood, breaks up an async method into multiple methods and puts values that appear to be on the stack onto the heap. (Of course, I'm over simplifying.) It just works, and it just works well.

But, in Rust, making something async implicitly moves what appears to be on the stack onto the heap. It can quickly become hard to reason about.

I wish I knew about techniques that this article describes. Maybe it would make my code easier? In my current hobby projects, I'm doing traditional blocking IO because it's not "worth it" to write async. (In comparison, in Node, async code helps avoid nesting callbacks within callbacks, and in C#, doing IO in async is a best practice.)

My hobby and main interest is multithreading, async, coroutines, parallelism so I love articles like this so thank you.

I am trying to design a solution that lets us have our cake and eat it to. I want multithreaded coroutines or multithreaded async executors by default. I am trying to design a server and runtime that is largely parallel, concurrent, efficient, easy to understand, easy to reason about, async and easy to read and maintain. I want:

* Tokio is not bad but I think a codebase that uses async rust requires a high level of skill, cognitive load and understanding. It's not straightforward!

* synchronous straight-line flow code is the easiest to read and follow

* promises and callbacks aren't easy to read or follow control flow

* making a single threaded program parallel after writing it is almost a complete rewrite

* work stealing thread pools solve the starvation problem, so they're good!

* IO shouldn't block CPU and CPU shouldn't block IO

* I am trying to design a syntax that is data orientated that means programs can be parallelised after writing them, so parallelisation comes for free

* we can use the LMAX Disruptor pattern for efficient cross-thread communication. I use a lockfree multiconsumer multiproducer ringbuffer in my programs.

I have an epoll-server which multiplexes clients/sockets over threads, this is more efficient than a thread-per-socket/client. I need to change it into a websocket server.

Imagine you're a search engine company and you want to index links between URLs. How would you model this with async rust and thread pools?

  task download-url
   for url in urls:
    download(url)

  task extract-links
   parsed = parse(document)
   return parsed

  task fetch-links
   for link in document.query("a")
    return link

  task save-data
   db.save(url, link)

`db.save()`, `download()` are IO intensive whereas `document.query("a")` and `parse` is CPU intensive.

I think its work diagram looks like this: https://github.com/samsquire/dream-programming-language/blob...

I've tried to design a multithreaded architecture that is scalable which combines lightweight threads + thread pools for work + control threads for IO epoll or liburing loops:

Here's the high level diagram:

https://github.com/samsquire/ideas5/blob/main/NonblockingRun...

The secret is modelling control flow as a data flow problem and having a simple but efficient scheduler.

I wrote about schedulers here and binpacking work into time:

https://github.com/samsquire/ideas4#196-binpacking-work-into...

I also have a 1:M:N lightweight thread scheduler/multiplexer:

https://github.com/samsquire/preemptible-thread

many many reasons and it's subtile and complicated

For one ironically it's just way easier to use, especially for less experienced programmers. In a task system like that it's just way easier to accidentally cause major issues when it's many tasks on one thread compared to multiple threads (at least with the guard rails rust provides for threading safety). On the other hand the performance overhead of by-default multi-threaded is just fine for a ton of use-cases to a point you could argue worrying about it is premature optimization.

Through it's important to state that a lot of this comes from the ecosystem around rust and not rust itself, as stuff like `LocalSet` shows you can have a non-multi threaded runtime and there is no reasons all the libraries you might use couldn't provide non multi thread safe versions. Some do. Just many decided that avoiding the performance overhead of being thread save isn't worth the maintenance overhead and additional foot guns it can provide.

Now naturally you can say "but node/deno/etc." but they are a completely different beast then "just" being not multi threaded by default. Like e.g. they don't have multi threaded code at all. Just single threaded code communicating through serialized messages (kinda). They also e.g. handle all I/O completely separated from you application code and don't have any non serialized communication between threads etc. etc.

Interestingly if you look at the design choices of the I/O Event loop (reactor) of tokio there are some conceptual similarities. Also there AFIK there is a company which is building something similar to the node model for rust using WASM.

I mean in the end the node-style approach is grate for building servers, but rust isn't just for building servers but much more general with much more low level use cases.

Now the main point where rust could improve quite a bit is to make it much easier to write a library which work very efficient with both cases. Code which crosses thread boundaries and code which doesn't. Currently you are often split between either implementing it twice, doing terrible unusable generics tricks or using tons of `cfg` (probably generated using macros/annotations) and hope no dependency accidentally imports the multi-thread feature when you don't need it. Non of this is really viable but it's a surprisingly hard problem. Currently the best idea I can come up for it is generic modules which make the "terrible to use generics tricks" usable, but it's probably not enough by a long stretch. (Even if a solution is found it might not work for Waker, even if it does you still might want sync/send Wakers in some cases.)

Imagine writing code for a computer game bot: move left, wait for enemy, attack enemy... You normally can't write it like this because it would block the rest of your program. Async allows you to go from "program sequential" to "bot sequential" for lack of better terms. If you are IO bound there is no need for threads. I often like to use one network thread and one GUI thread to keep things separate, and to prevent the occasional blocking in one to cause latency in the other. You just need to have a method to post calls to the other event loop. Works well in Python, as well with Qt-based apps.

C# on the other hand made the same "mistake" of being very general. I think you can even have a coroutine suspend on one thread and wake on another. It looks like you switch threads in the middle of a function.

I guess that is useful when you want to write performant multithreaded servers. I just want to write easy sequential code without worrying about locks or state machines.