Request Coalescing in Async Rust
fasterthanli.me
fasterthanli.me
I understand this was more of a side point as an introduction, but it's a very quick dismissal of blocking I/O that I've been seeing a lot. Modern threads + blocking I/O is much more performant than most people realize, often yielding better throughput than async due to reduced syscalls and other factors. Async is important when you are in an extremely latency constrained environment and/or need to prioritize tasks, but in other places, not so much. How much does increased stack memory usage really matter to a real server?
The situation for a purely backend service (behind an existing reverse proxy) might very well be different — but as others have mentioned, the Rust http ecosystem has solidified around async, so even if you could get away with 1 connection = 1 thread, you might not want to, just because of where the ecosystem is.
One thing I didn't mention is that epoll isn't even state of the art: there's a lot of work going on around io-uring and thread-per-core runtimes nowadays, which I'm following rather closely because again, at my day job it does matter!
io-uring is definitely a game changer, but it isn't async specific!
- people lack understanding of fundamentals, thus buying into the hype, and are bad at weighing the trade-offs because they often can't see the downsides, especially regarding complexity
- engineers are curious and want to play with $latest_tech, and then construct a post-hoc justification for it to which they are sometimes unaware themselves
Or put differently, its either incompetence or mismatched incentives (principal-agent problem). There is some justification for playing with new tech as in "it helps motivation and retains talent", but sometimes the resulting complexity is far out of proportion.
That being said, tech ecosystems have different cultures, and I can understand why Rust is the way it is. It attracts idealists that tolerate complexity in the pursuit of the optimal solution. They have already produced more useful software than eg Haskell, and this is ultimately the yardstick of success. Lets see how Zig will do in that regard.
> those are pointless for an internal backend with at most 10 concurrent connections
I'm also arguing that the benefit of async over threads at 10 _thousand_ concurrent connections is still unclear. Yes memory usage will be less, but memory is cheap. Context switching overhead may be less, but a lot of it is still there because of readiness-based I/O. We don't see new, high-scale systems built on blocking I/O perhaps not because they can't be, but because async has become the norm, at the cost of _a lot_ of complexity.
Does anyone else think the Rust HTTP ecosystem is becoming increasingly fragmented? I can't keep track of which library is best suited for common operations. Is it a web framework built on hyper? Should I pay attention to tower middleware? Where does tracing fit in?
- There was a split between tokio and async-std asynchronous executors, but the ecosystem now seems to be coalescing back around tokio.
- There was weird split where Hyper was the de facto http library, but the best web framework was actix-web which wasn't based on hyper. But now there is Axum, an official Tokio project that is good enough to generally recommended for all web projects, and looks to be the project with momentum going forwards.
- Tracing is really the only game in town when it comes to asynchronous logging, and again is part of the Tokio project.
Tower is a bit of a weird one. It's a very general middleware layer which I think does actually provide a very good abstraction. But that abstraction is quite complex, and learning it is in many cases trickier than doing an implementation from scratch. I suspect it might come to play a bigger part in the Rust ecosystem at some point, but for now I'd ignore it.
I don't write Rust professionally, but it was a bummer seeing that this seems to be a place that was figured out (painfully) in ecosystems used heavily for web development--Javascript and Elixir have their own Rack equivalents[2][3]. I hope that Tower plays a similar role to unify the library ecosystem in Rust.
1. https://github.com/rack/rack
actix-web, like hyper, is built on tokio, so I'm not sure why you see this as a weird split. The underlying HTTP server is not something you generally interact with. actix-web even _uses_ hyper (the h2 crate) for HTTP2.
And its begging the question "Is this what I want, or should ditch these, and build a new one using threads". I am not sold on the async paradigm. Some of the Rust OSS community uses it on embedded too, but to me, code using interrupt handlers (or RTIC) is more intuitive.
Immediately on viewing the Hyper guide and hello world, the program structure is messier than a normal Rust program. Colors propagate through the program. I am giving this a shot, but am not optimistic this will be cleaner than threading.
The point of async is that at scale threads start becoming expensive. If you don't have high performance requirements under heavy load, async is largely unnecessary.
For whatever it's worth, we are hiring for a non cryptocurrency Rust job, haha.
1. write a 'Hello World' HTTP server to show how Tokio's async-Rust primitives work for epoll-based I/O;
2. show how the tracing crate ecosystem works to inspect and debug async-runtime tasks;
3. play with converting traces to OpenTelemetry, tokio-console, etc. for different views of the collected data;
4. play with hyper and axum to build a simple HTTP service that makes an upstream HTTP request to the YouTube API;
5. cache results by ID so it can shared across all request-handlers;
6. add 'request coalescing' to prevent multiple concurrent requests from making upstream requests in parallel;
7. fix a bug in the request-coalescing implementation that caused coalesced requests to hang when the in-flight request raised an error.
One word of caution about using any request-coalescing pattern like this on a highly-loaded server: in the case where every request fails to populate the cache (e.g., timeout / not found), you end up in a situation where all requests end up _serialized_ and can quickly lock up a server under constant load. You need to either add a circuit-breaker, or otherwise ensure that _something_ always gets cached in any otherwise-uncacheable edge case (e.g., hit-for-pass in Varnish [1].)
[1] https://info.varnish-software.com/blog/hit-for-pass-varnish-...
I enjoy your long reads but picking a post back up after pausing can be tricky.
That's Amos' brand, alright! ;)
(<3 Amos)
Amos writes some amazingly readable, incredibly long posts. Such a treasure.
EDIT: Actually I'm guessing a lot of it comes down to encoding the HTTP responses, now that I think about it.
Of course if you reach a point where the only thing left to do is implement http/2, that... is no longer an evening project
A minimalist, custom web server can approach 10m req/s as seen on the Techempower Benchmarks.
> As the popular saying goes, there are only two hard problems in computer science: caching, off-by-one errors, and getting a Rust job that isn't cryptocurrency-related.
I'm hiring Rust engineers for the future of virtual production. Kids at home are going to leapfrog Disney and make their own Star Wars.
We have some toys that might look silly now, but so much in the pipeline once all of this begins to mature.
- https://FakeYou.com is viral on social media. (Please excuse the service failures as we just reached 1M unique DAUs. Creating an account and logging in boosts queue priority such that your requests will render in under a few seconds.)
- https://create.Storyteller.io does Twitch TTS and deepfaked cheer emotes, has a robust rule based system, and helps creators monetize and engage their audience.
- We also have motion capture, volumetric capture, voice conversion, virtual sets, etc. But I'm so tapped out with the growth that these haven't been spun off yet apart from internal use and close work with a few creators.
Silly toys that don't look like much now, but incredible promise, incredible traction, and signs of so many monetization channels. (Custom voice cloning, paid services for Twitch creators, broadcast studios, hosted rendering, etc. etc. etc.)
I'll hire contract, part time, remote. Post funding (working on this now), huge equity, salary, etc. are in line. But I'm more than happy to draw down my last startup exit to fund this.
Unlike working on plumbing and glue code, this is actually a ton of fun and touches on so many of the most exciting parts of CS. ML, computer vision, audio processing, graphics, ...
Sorry if hustle posts like this break the rules, dang! (The article led so brilliantly I had to say something)
Contact info in my profile, or jump in our Discord.
Great line. I've been thinking of using Rust (in addition to its merits) as a hiring carrot, to help attract some of the best multi-skilled software engineers.
I've seen this work very well with Common Lisp and Scheme in the past.