Principles for Fast Tokio Applications | Hacker News Reader

Principles for Fast Tokio Applications

dial9-rs.github.io

249 pointscarllerche66 comments

"Be careful with mutexes" is good advice, but I'm surprised it doesn't explicitly call out the various channels that tokio provides as alternatives (detailed here: https://docs.rs/tokio/latest/tokio/sync/index.html). There are a variety of options that fit different use cases, and you don't even need to enable the runtime feature to use them (e.g. if you want to do a single check for completion rather than await). I'd estimate that at least half of the bottlenecks I've seen with mutexes when using tokio could have been avoided by not even using a mutex at all and instead passing the data that's truly needed across different tasks with some type of channel.

The other trick I've used a few times that's a bit hacky but can get the job done is when reading a snapshot of the data under a mutex is enough without needing to prevent other changes; if that's the case, you can just clone the data and drop the mutex to allow other uses move forward at the cost of the data potentially being stale.

(I am OP) Both good call outs. Will update the article to include them

Awesome! I was pretty confident you already were aware of both of those based on the level of knowledge needed for everything else in there, so I mostly was mentioning them here in case some people here might find them useful. Adding them in for others is even better though!

Tasks and channels is the way. You can get something that feels like programming a real preemptive concurrency model like BEAM languages or golang but with minimal overhead.

When you send a message in Erlang, nothing the recipient does with the message impacts anything on the sender side. That's good!

In principle, they could have used something like copy-on-write for this, but in practice they really just make a copy of the bytes.

Alas in Go, when you mutate what you received on a channel, you mutate the object the sender might still be holding. That's pretty annoying. It gets worse, because Golang has no way to declare something as `const` (like in C) nor that you are holding an immutable borrow (like in Rust). So you need to rely on conventions and perhaps a linter.

Slighty less of a tangent: task and channels and software transactional memory (STM) are all great. I see mutexes as more of an implementation detail that you can use to implement these higher level abstractions (but they aren't the only way).

I honestly stopped taking Go channels seriously when I found out that reading from a closed channel is indistinguishable from reading the zero value, but writing to a closed channel will panic. I don't want to have to use booleans and write `true` in order to ping another task and have it tell the difference between getting pinged or being hung up on, and I really don't want to have to architect things so that I need to have my tasks know when the other side of a channel is closed before writing to it (or catch panics instead of using regular error handling) because that feels like it defeats the whole purpose of not needing direct knowledge of the state of the other side. It genuinely seems like those design decisions basically came from a desire to use special operators on channels rather than just having `send` and `receive` methods (or worse, to intentionally avoid "tedious" error handling/optional checks), which is mind-boggling.

I hadn't even considered the implications of sending a reference over a channel, but I'll add that to the existing reasons I have to never want to touch Go again.

Historically, Go was also really weird in having a very arrogant design: the language designers allowed themselves a lot of generic operators and structures (like arrays, map, channel, the 'make' function etc), but they were considered taboo for the hoi polloi which had to make do with the equivalent of void* pointers and runtime casting (the empty interface shenanigans).

I say historically, because they got generics a while ago.

Yeah, I honestly was surprised they did end up eventually getting generics. My brief stint with it ended a while before then, but my impression at the time was that they were bizarrely insistent on trying to come up with a design from first principles rather than taking the time to understand the literal decades of designs that came before. It was impossible for me to tell the difference between their position and "we don't like Java generics and are either not aware of languages like ML or are too arrogant to read about them".

Give a task ownership of some state, communicate through channels and suddenly a lot of locking just disappears from the design

I've long pined for an ergonomic way of defining actors in Rust. I feel like there must be some way to abstract things that doesn't leak a bunch of a details into the mental model around how to think about starting/stopping/communicating between actors, but every time I've tried to figure it out (or use a solution someone else made) it ends up being way more complicated than it feels like it needs to be, and not worth it over writing a bunch of manual tasks wrapping private structs that have a bunch of channels in them. I feel like I've tried everything I could think of in terms of API design to make this work in a way that doesn't require users either having to learn a bunch of bespoke rules for the implementation or spend a lot of extra effort on manual boilerplate, including some very wacky things (like a macro where you pass a name and a function and it defines you a new macro with that name for spawning the actor task), but nothing ends up like I'd want.

Yeah the Actor model just doesn't fit well with Rust without feeling like a DSL. Tokio works well because Rust's ownership model overlays with tokio nicely. The language is already doing the hard data safety stuff for tokio, tokio just adds a lot of conveniences.

Erlang and Go make their concurrency models work because it is embedded into the fabric of the language. Neither cares about zero cost abstractions or minimal runtime (and runtime transparency). Both languages accept that there is a runtime that the developer cannot fully control as part of the deal for their concurrency models.

For Rust to have this, you have to break assumptions Rust developers have about writing Rust code. You would effectively be writing a runtime in Rust and then code that uses this Actor library would essentially run on it. But it wouldn't feel right because it would look like Rust code but feel like something else. That sort of heavy framework stuff doesn't mesh super well with Rust even if the language is capable of it.

I guess in the framing you're describing, the issue I ran into is that Tokio still just isn't anywhere close to as opinionated as a runtime as the other ones you mention. Even just taking message passing as an example, Go gives you one channel, and that's what you have to use for everything; Tokio gives you a single use channel, both a bounded and unbounded channel for an arbitrary number of senders to a single receiver, and two separate multi-producer-multi-consumer channels with different properties. I could make a bunch of opinionated choices of my own about how this should work, but at that point I'm adding a bunch of potentially leaky abstractions on top of a runtime that already isn't core to the language (and as the blog post indicates, there's a learning curve about how to get the most out of it).

Why use mutexes (mutices?) over semaphores?

Aren't semaphores a more fundamental primitive that is trickier to get right? Otherwise, the mutex guards in Rust are very ergonomic.

Semaphores are more general for sure. The common semaphore is just a counter, when you lock it you decrement the counter by one atomically. Therefore a mutex is just a semaphore with a limit of 1. I wouldn't say they're much trickier to get right, maybe just less frequently applicable in e.g. general web io tasks.

I suspect the reason that semaphores aren't as common in Rust specifically is that once you go above one reference to something, you stop being able to mutate it safely. Mutexes work because "one thing writing to this means nothing else can read it" doesn't go against the borrow-checking grain, but as soon as you want to allow two references to something, you can't write to it safely, so it only works when you want to delay processing of non-shared data (which as you mention is not particularly common) or read-only processing (at which point you probably just want an RwLock to get shared read-only concurrent access).

I think channels deserve more emphasis here too, especially because they change the architecture rather than just swapping synchronization primitives

All of the significant server applications I have encountered in the industry have suffered from the same problem, which surprised their authors but seemed obvious to me: the application was spending the majority of its CPU time doing meta-work like entering and leaving epoll, stealing work from itself, etc. There are principles for writing Tokio servers and these are good points in the OP but I think they are little-known and too easy to violate.

I can't say I'm surprised when I see the 100+ function stack traces that Axum built on Tokio produces.

Before you say Axum is "holding it wrong" the project lives under the tokio-rs GitHub org.

Note that most of those end up getting inlined in practice

Do you have any references for these principles for writing Tokio servers? Or just a high level summary of what best practices look like?

When you're at a point of tuning Tokio, consider taking a look at ef_vi/DPDK + SPDK

Do you have any resources worth referencing on this? I assume this isn't something that works with tokio more of a replace tokio?

I don't think there is a ton of overlap. tokio is appropriate for general userspace apps, ranging anywhere from a CLI, GUI, API or web app. DPDK and SPDK are specialized fast paths for building network data paths and storage solutions that come with tradeoffs: DPDK uses poll mode drivers, outside of the operating system, which have various implications including busy waiting and taking over the interface. That is why DPDK is fast, no kernel/userspace context switching and copies, and the drivers are tuned for the polling model. But it's not a general purpose building block.

Fwiw with ef_vi you have full control over the event queue - you don't need to busy-spin it, you can choose whatever strategy you prefer.

> tokio is appropriate for general userspace apps

Yep, and for those I wouldn't recommend it. But tokio is also widely used in performance-critical infrastructure and web services. For those I'd say it can definitely be worth taking a second look at kernel bypass.

ef_vi is a solarflare proprietary feature which is now a support product, AMD moved on to Pensando. Once you move away from busy poll, you rapidly lost grounds to use DPDK. The PMD is a deliberate design to elide latency and lower interconnect taxes like PCIe traffic and cache/memory bandwidth by batching queue maintenance, that is the bargain made with a PMD. The field opens to OS native fast paths which have fewer downsides outside of that niche. Application developers are rarely concerned with this because it's far from where the bottleneck is for them.. a web service is rarely primarily a data mover, while a proxy is. Tokio has more in common with Golang than something like DPDK.

For a true high performance you should use thread busy-spinning, CPU pinning and SPSC/MPSC ring buffers.

It all depends on what you are doing. I do embedded with strict realtime requirements. CPU pinning would not be an option. I have also done software that should use as little resources as possible (but still be quick) to coexist with other software on the same hardware.

All of these are different, valid, meanings of high performance. You need context. An interactive IDE is yet another thing that needs to be high performance in yet another way.

Also, using 100% CPU without a good reason can cause thermal throttling that makes it slower for the sections that actually need 100% CPU

Spinning is covered in the article.

One great use of agentic coding is being able to add and very granular tracing instrumentation to help with these sort of optimizations.

Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.

If you’re not using eBPF to trace your app you’re doing it wrong.

The low cost of eBPF tracing is another myth.

1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it's a far better path to take than in-app telemetry.

Doesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.

One legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.

I can get an LLM to trace my incomprehensible Tokio application which was also written by an LLM, which is why I don't understand its behavior. Truly the future we were promised.

I guess you should adopt RFCs or ADRs to help clarify the Tokio application, like this https://github.com/brunoarueira/thoth-mesh/tree/main/docs/ad.... This project is vibe coded, but I had put the effort to create issues, roadmap and ADRs, so later I can understand the project without going deep on the code!

Reaching source control is fine as long as there is a compile time flag to disable the whole thing, which tokio-tracing does

That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload.

You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range.

The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.

I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe/uretprobe is 100x more expensive than XRay, at a minimum. HotSpot's JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio.

1: https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?

Not even an analog: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/...

It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces.

afaik, C/C++ have the same problem.

The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.

The "C++ community", if it even exists, barely believes in sharing code let alone any library being "dominant." They'd have to agree on a build system first, after all.

But honestly that's a mischaracterization of the situation in Rust. Tokio is popular for networked service backends. If that's the wheelhouse you're in then yea it might look "dominant."

You don't need a build system to share code.

You can share with header files and respective (shared) object files regardless of the build system you're using. Likewise you could just share the source. None of this needs a build system.

I was just being a bit sardonic because the C++ ecosystem is so fragmented that something like tokio couldn't really exist. It would be one of three executors in boost, abseil, or folly, and you would never see the kind of downstream ecosystem build on top of them because C++ shops are allergic to external dependencies.

Any organisation that cares about security should be allergic to external dependencies, that is why companies like Nexus and JFrog exist, with companies paying to keep internal repos infrastructure in shape.

One just doesn't install willy nilly from the Internet into the CI/CD pipeline.

Well, they do, and then spend a few late nights when there is a bunch of CVE to fix.

Using an artifactory instance as the origin is not functionally different from installing something 'willy nilly' from the internet. It addresses a narrow range of threats while predominantly being more reliable and faster than public repositories.

It doesn't fix the actual problems with C++, which is that it's significantly more difficult to get and use external dependencies because of the compilation and linkage model of C++ libraries.

If C++ were as easy to build and link as modern programming languages you'd see the same kinds of tools as cargo, and the same kinds of ecosystem evolution as rust, like tokio. But you don't, because C++ code sucks to build, package, distribute, update, and reuse.

(I'm aware/have used conan/meson/vcpkg/etc - doesn't change my opinion).

It surely does, because in most companies that care about security it isn't a mirror, rather the only third party packages that developers are allowed to use beyond the standard library.

Additionally, making new packages available for consumption requires approval from IT and possibly legal, before they become available for consumption.

What is hard is people educated in scripting languages not wanting to learn about toolchains.

The moment Rust depends on other programming languages, we get a build.rs spaghetti file, depending on the knowledge of those writing it, or people throwing away Cargo altogether, and replacing it with Bazel, buck2 and co.

Of course we do, it is done via OS package managers, commercial libraries and SDKs.

More recently, via vcpkg and conan.

Huh, it seems xray puts blank trampolines all over your binary? That sounds pretty nifty but I would expect it to be pretty language agnostic, ish? Adding support should be doable for Rust as well, right? Anyways, pretty nifty.

I am by no means an expert, but I've recently improved performance for some code and used tracy. They have rust bindings as well. It's pretty cool and it seems to be low overhead. Wonder if I can couple it with something like xray? Tracy is more the tracing library + tracing interpretations/aquisition tool.

Edit: apparently rust already supports xray natively on the nightly.

Just curious, why? Is this true even if you did something like a per-CPU histogram that uses atomic ops to increment?

If you have a per-cpu metric there would not be a reason to use atomic instructions to mutate it.

In general your unpinned userspace threads will hit the same CPU 99.99% of the time, but not 100%.

Sure. You get the pointer, you lock the mutex, 99.99% of the time that is uncontended, then you set all the metrics and release it.

Taking the mutex uses (uncontended) atomic ops.

Was this in a specific application? I wouldn't necessarily expect that histogram to be particularly bad for most applications.

Reading the clock every time you jump into a closure is in fact incredibly wasteful, and is exacerbated by chopping work up into tiny chunks for questionable reasons.

I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a one-off test. E.g., I have some code floating around which burns a pinned core on increasing a counter, with a little wrapper code around grabbing real timestamps at the beginning and end of a session and converting between the two units of time. It's helpful when microbenchmarking a very small unit of code as it actually behaves in a larger program (not perfect -- obviously tweaks the icache and pipeline behavior at a minimum -- but no measurement has zero tradeoffs, and you're always choosing which set of tradeoffs you prefer). An LLM can quickly instrument the call path I care about while I study this or that intervention. The ability to bang out a large amount of throwaway code is delightful.

When working on 20ms audio frames, Trusting MissedTickBehavior::Delay is not enough by itself. If you miss a tick, frames pile up. If you don't want that, you should drain all accumulated full frames on every single tick. Otherwise, a single missed tick can cause permanent latency.

One thing I appreciate here is treating scheduler fairness as something you spend, not something you get for free

What the hell is Tokio? Articles mentions it like once I was expecting some programing principles from Japan

https://github.com/tokio-rs/tokio

To further explain. Rust doesn't provide a runtime/framework for async/await, you have to bring your own. Tokio is (I believe) the most popular async/await framework for rust.

Fast Tokioo, drift, drift, drift!

Reply on news.ycombinator.com