How Rust Lets Us Monitor 30k API calls/min
blog.bearer.sh
blog.bearer.sh
I know this comes across as snarky but it really worries me that contemporary engineers think this is a feat worthy of a blog post. For example, take this book from 2003 [1] talking about Apache + mod_perl. Page 325 [2] shows a benchmark: "As you can see, the server was able to respond on average to 856 requests per second... and 10 milliseconds to process each request".
And just to show this isn't a NodeJS vs Rust thing, check out these webframework benchmarks using various JS frameworks [3]. The worst performer on there still does >500 rps while the best does 500,000.
It's 2020, the bar needs to be much higher.
[1] https://www.amazon.com/Practical-mod_perl-Stas-Bekman/dp/059...
[2] https://books.google.com/books?id=i3Ww_7a2Ff4C&pg=PT356&lpg=...
[3] https://www.techempower.com/benchmarks/#section=data-r19&hw=...
That sounds like a decent amount of work for a service, and without more detail it's very hard to say whether or not a given level is efficient or inefficient (we don't know exactly what was being done; we can assume that they're using pretty small Fargate instances though since the Node one came in at 1.5G). They also give some number; 4k RPM was their scaleout point for Node (that's not necessarily the maximum, but the point they felt load was sufficiently high to warrant a scaleout; certainly, their graph shows an average latency > 1 second). Rewriting in Rust, that number was raised to 30k RPM; 100 mb of memory, < 40ms average latency (and way better max), and 2.5% of CPU.
Given all that, it sounds like, yes, GC was the issue (both high memory and CPU pressure), and with the Rust implementation (no GC) they're nowhere near any CPU or memory limit, and so the 30k is likely a network bottleneck.
That said, while I agree that sounds like a terrible metric on the face of it, with what data they've provided (and without anything else), it also sounds like it may be due to they're just operationally dealing with very large amounts of traffic. They may want to consider optimizing the network pipe; not familiar enough with Fargate, but if it's like EC2, there may be a sizing of cpu/memory that also gives you a better network connection (EC2 goes from 1 GBPS to a 10 GBPS network card at one instance type)
5+ years ago I wrote a real-time transcoding, muxing streaming radio service that did 5000 simultaneous connections with inline, per-client ad spot injection (every 30 seconds in my benchmark). Using C and Lua. On 2 Xeon E3 cores--1 core for all the stream transcoding, muxing, and HTTP/RTSP setup, 1 core for the Lua controller (which was mostly idle). The ceiling was handling all the NIC IRQs.
While I think what I did was cool, I know people can eke much more performance out of their hardware than I can. And I wasn't even trying too hard--my emphasis is always on writing clear code and simple abstractions (though that often translates into cache-friendly code).
At my day job, in the past two months I've seen two services in a "scalable" k8s clusters fall over because the daemons were running with file descriptor ulimits of 1024. "Highly concurrent" Go-based daemons. For all the emphasis on scale, apparently none of the engineers had yet hit the teeny, tiny 1024 descriptor limit.
We really do need to raise our expectations a little.
I haven't written any Rust but I have recently helped someone writing a concurrent Rust-based reverse proxy service debug their Rust code and from my vantage point I have some serious criticisms of Tokio. Some of the decisions are clearly premature optimization chosen by people who probably haven't actually developed and pushed into production a process that handles 10s of thousands of concurrent connections, single-threaded or multi-threaded. At least not without a team of people debugging things and pushing it along. For example, their choice of defaulting to edge-triggered instead of level-triggered notification shows a failure to appreciate the difficulties of managing backpressure, or debugging lost edge-triggered readiness state. These are hard lessons to learn, but people don't often learn them because in practice it's cheaper and easier to scale up with EC2 than it is to actually write a solid piece of software.
I've seen places hit ulimit limits...I've also seen places hit port assignment issues, where they're calling out to a downstream that can handle thousands of requests with a single instance, so there are two, and there aren't enough port identifiers to support that (and the engineers are relying on code that isn't reusing connections properly). Those are all things worth learning to do right, agreed, and generally doing right. I'm just reluctant to call out someone for doing something wrong unless I -know- they're doing something wrong. The numbers don't tell the whole story.
Here's what they're doing:
> Now, when the Bearer Agent in a user's application sends log data to Bearer, it goes into the Envoy proxy. Envoy looks at the request and communicates with Redis to check things like rate limits, authorization details, and usage quotas. Next, the Rust application running alongside Envoy prepares the log data and passes it through Kinesis into an s3 bucket for storage. S3 then triggers our worker to fetch and process the data so Elastic Search can index it. At this point, our users can access the data in our dashboard.
Given their goal and their problems with GC I can tell you right off the bat probably what's the problem with their various architectures from day 1--too much simplistic string munging. If your idea of log ingestion is using in-language regex constructs to chop up strings into pieces, possibly wrapping them in abstract objects, then its predictable you're going to have GC issues, and memory bandwidth issues in general, and poor cache locality in data and code. But 99% of the time this is how people approach the issue.
What a problem like this cries out for is a streaming DFA architecture, using something like Ragel so you can operate on streams and output flat data structures. You could probably implement most of the application logic and I/O in your scripting language of choice, unoptimized GC and all, so long as you're not chopping up a gazillion log lines into a gazillion^2 strings. The latter approach will cause you grief in any language, whether it's JavaScript, Java, Go, Rust or C. The number of objects per connection should be and can be a small N. For example, at 10 distinct objects (incoming connection object, log line, data structure with decomposed metadata, output connection object, etc) per connection times 500 connections, that's 5000 objects per second. Even Python's and Ruby's GC wouldn't break a sweat handling that, even though internally it'd be closer to 10 * (2 or 3) objects.
Here's a big problem today: nobody writes their own HTTP library or JSON library; everybody uses the most popular ones. So right off the bat every ingestion call is going to generate hundreds or thousands of objects because popular third-party libraries generally suck in each request and explode it into huge, deeply nested data structures. Even in Rust. You can't optimize that inefficiency away. No amount of fearless concurrency, transactional memory, fastest-in-the-world hashing library, or coolest regular expression engine can even begin to compensate. You have to avoid it from day 1. But if your expectations about what's possible are wrong (including how tractable it is with some experience), it won't even occur to you that you can do better. Instead, you'll just recapitulate the same architectural sins in the next fastest language.
Emphasis added. I don't disagree with you that they may be doing something inefficient; I'm just saying, I don't -know- what they're doing, so I'm disinclined to judge it.
I do know that, again, in Rust, whatever bottleneck they're hitting is neither CPU nor memory, despite the seemingly low throughput, which does imply that what you're proposing isn't the bottleneck in that implementation.
Microsoft, Apple, Amazon, Oxide, Mozilla, Dropbox and CloudFlare would like a word...
I'm aware of at least two ongoing projects they're writing in Rust, which I obviously won't comment on in detail publicly.
Tier 2 means it requires approval. That is massively different from "forbidden". There is, for example, a Tier 3 list - Java is on there, some other languages, they're discouraged a lot more strongly than Rust though you'll still find them in some parts of the codebase (primarily acquired code from what I recall). Approval is only required for business/ product dev - non product teams can very easily write rust, at least one is currently doing so.
Tier 2 means it has internal library support, such as the communication library. That means there is, at all times, active rust development - and of course this is true, the most critical parts of Dropbox are written in Rust, and they rely on those libraries.
You can also very easily find out about the existing projects being written in Rust. Go ask the rust-lang channel. Last I saw, maybe 6 months ago, Rust was being used for another major component of the product - obviously I'm not commenting publicly on that further.
It is definitely not the case that any randomly selected backend engineer at that company can just pick up Rust and solve any problem with it, because the Tier 2 status is used as a cudgel to stop most such efforts.
I agree with your point about political issues - the entire tier list, specifically even Rust being tier 2, was political. I saw stupid shit like that too many times at Dropbox.
But it is simply a fact that new projects are being built in Rust, regardless of the political aspects of why that is the case despite it being tier 2. "Forbidden" does not convey the state of things - everything else you've said is agreeable.
500 requests per second is what I would expect of a default PHP + Apache installation on a small Ubuntu server.
I too have a hard time grasping whats special here. For example I saw cached Wordpress setups handle 400 to 500 requests per second. And Wordpress isn't known for performance even with caching plugins.
As someone who has been developing on the BEAM for long time now, it usually sticks out like a sore thumb any time I see Elixir/Erlang paired with Redis. Not that there is anything wrong with Redis, but most of the time you can save yourself the additional Ops dependency and application network hop by bringing that state into your application (BEAM languages excel at writing stateful applications).
In the article you write that you were using Redis for rate limit checks. You could have very easily bundled that validation into the Elixir application and had for example a single GenServer running per customer that performs the rate limiting validation (I actually wrote a blog post on this using the leaky bucket and token bucket algorithms https://akoutmos.com/post/rate-limiting-with-genservers/). Pair this with hot code deployments, you would not lose rate limit values across application deployments.
I would be curious to see how much more mileage you could have gotten with that given that the Node application would not have to make network calls to the Elixir service and Redis.
Just wanted to share that little tidbit as it is something that I see quite often with people new to the BEAM :). Thanks again for sharing!
If you're load balancing on IP hash you can set each instance to limit at full rate and not worry about it.
Shared state in rate limiting becomes a bottleneck very quickly. If you're trying to mitigate spam/DDOS you could easily get 100,000 requests a second. You're going to max out your shared state db way faster than 10gig lines
I have never used this product so just speculation. But I imagine there is some sort of auth token that valid agents send to tell Bearer that this is a valid/invalid request so that things can be trivially rejected to mitigate a DoS/DDoS to an extent.
Rust is closer to a super fancy C, compiles natively and was made with a heavily focus on certain types of memory safety.
F# has syntax more like Haskell/ML, and is compiled to a bytecode instead of an executable. It runs on .NET and everything that entails.
I would imagine so. Rust doesn't support tail call optimization, and variables are immutable only by default.
let a= [1, 2];
a.iter().filter(|x|..).map(|x|...)
https://doc.rust-lang.org/std/convert/trait.Into.html
This is THE feature I wish other langs copy. Combined with serde:
https://github.com/serde-rs/serde
Is possible to cut a lot of boilerplate related to data transformation (that is the main task in my case).
An article published today, addressing that exact point: https://tim.mcnamara.nz/post/621040767010504704/spend-your-n...
This article should mention that rust isn't close to ready when it comes to web backends. As much as I love Rust, if I were running a startup or even a decently sized company I would always choose Rails. Now -that- is boring, old... and mature technology. Certain components could get re-written in Rust, certainly, but there's no reason to ignore a mature ecosystem from the start.
Actix-web works just fine. They got a new maintainer team involved that has been spending some time getting rid of all the insane unsafety that was in the code before.
Rails is mature, robust, and has a huge ecosystem with rubygems. Rust (when it comes to web stuff) is not. "it works" does not pass my litmus test. With Actix/Warp I have to implement stuff by hand that either comes by default with rails or already exists in a gem.
I like Rust but I'm not a zealot. People way overestimate performance when they barely have any traffic to begin with as a small startup or even a medium sized company.
You could even use rails, and use Rust to write ruby gems instead of going with actix/warp/etc.
> insane unsafety
This was overblown. Yes, the author didn't respond appropriately, but unsafe isn't inherently dangerous. This is a stupid misconception within the Rust community and caused a lot of unnecessary drama around Actix.
Ultimately in a startup it comes to what people are comfortable with. I personally would use Rust for something like this because I am comfortable with Rust. It's a perfect use case for it as well, imho.
If you want to see an example of fully zero-GC Java, you can check out QuestDB on Github [1] - Disclaimer I work for QuestDB.
I don't know that it's actually possible in the general case, as Java's support for value types remains wholly insufficient. IIRC the ixy folks never managed to remove all allocations from the java version.
Not sure why they'd consider Java, given that concern.
The other option is to avoid allocating memory at all, which you could do in C/Rust but also in Java. The vast majority of shops given the choice for low/no allocation performance use Java anyways (HFT)
"The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. Java and trying to minimize allocations, vs C/C++, I'd probably choose Java too. Java trying to minimize allocations (no way to guarantee you've done it right), vs Rust (which does guarantee no GC)...I'd probably pick Rust.
You could malloc() and free() so much that the code doesn't have time to do anything too. And these operations aren't bounded in time either. Just using C doesn't save you from memory allocation, its just done manually instead of automatic. In every system I've worked on you won't have this kind of GC pressure unless you do something profoundly bad
> The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed.
true
With GC, I've definitely had plenty of times where GC created very noticeable pressure on the running system. Nothing quite as bad as they describe here, true, but that would probably predispose me to ditching GC if I saw that kind of behavior, too (since it's so unlikely that either my case is truly deviant, or my devs have done something very wrong, and neither of those is something I want to bet on being able to fix).
With how low the pause times are with Java's new collectors (categorically not longer than malloc()), the only faster alternative is not allocating at all. Which you can do in many languages.
I agree your point the memory allocation and GC is expensive, but that's something you see in any language when allocating. It's not unique to the GC. You either pay the price with GC or malloc()/free(). Its the same price but one is much easier to deal with than the other.
If pause times lower than 10ms are important you could write your app in Rust which makes it much easier to systematically avoid allocations using borrowing.
I guess my point is that the new collectors make Java memory management roughly comparable to C, and if you care so much about performance that allocation costs are important, you should just use a language like Rust where its easy to avoid allocation completely. The performance gap between allocating in C vs Java is, for practical purposes, gone.
Go collects VERY frequently to keep average pause times low, which hurts throughput. It also has pathological worst-case pause times, which is what the author ran into.
My suggestion of Java is just because its been used for high performance REST for decades. Rust has Actix and a few other frameworks, in Java you have 20+ options. Its a lot easier to get something off the ground when you're 99% sure you won't have to build anything except some glue code. Yes, Rust is faster, but is 30% better performance worth treading the wilds while you're trying to keep a startup afloat, I don't think so personally.
Using Java with Shenandoah or ZGC collector would directly handle their GC issue with rather boring technology
Did you try using the kinesis REST API directly: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_...
Of course, this has its own issues, but I've seen it done in e.g. Go before. Its likely you'll inevitably end up with leaks, but if your service is fungible and can tolerate restarts, basically what you're doing is moving the "GC Pause" to be a "Container Restart" pause, which may be slower, but would happen less often. Some languages have ways to manually call the GC (Node is not one of them, afaik).
I've only seen this done in Go :)
[0]: https://v8.dev/
[1]: http://nodejs.org/dist/latest-v12.x/docs/api/inspector.html#...
[2]: https://github.com/davidmarkclements/0x
[3]: https://github.com/vmarchaud/openprofiling-node
PS: Disclamer i wrote one of tool i mentionned above [3]