Data-driven performance optimization with Rust and Miri
medium.com
medium.com
I get the feeling that a lot of Rust projects claim to be "blazingly fast" just because they are written in Rust, and not because they've made any attempts to actually optimize it. I rarely see any realistic benchmarks, and the few times I've looked deeply into the designs they are not implemented with execution speed in mind, or in some cases prematurely optimized in a way that is actively detrimental [1].
Personally I think it's because so many of the new Rust programmers are coming from scripting languages, so everything feels fast. I don't have any problems with that, but I'd advise anyone seeing a "blazingly fast" Rust project to check if the project has even a single reasonable benchmark to back that up.
https://github.com/rust-lang/hashbrown
https://github.com/briansmith/ring
https://github.com/rust-random/rand
Lot of Rust programmers also coming from C, C++, and Go btw.
And admittedly I do agree that uutils is not really a good example.
Surely there are some people that advertise their projects as fast but actually aren't, or at least aren't in some cases, whether they be written in Rust or anything else. (Even in ripgrep's case, of which I have published benchmarks, GNU grep is still sometimes faster.) But are there any substantial programs with such false advertising? It's hard to qualify what "substantial" means, but perhaps one could start with "is available in Debian" as a starting point.
Although it seems you aren't necessarily complaining about false advertising, but rather, missing benchmarks. Most programs, regardless of advertising, don't publish carefully curated benchmarks. I myself have advertised Rust's regex crate as "fast" (albeit not "blazingly fast," I am not one for such flowery language), but I have never published any benchmarks. Of course, benchmarks exist and others can run them, but they are more for internal development than public consumption.
Much benchmarking proceeds by an ad hoc nature. I'm not even aware of such venerable programs as GNU grep published benchmarks either, for example.
To be clear, I agree it would be nice for programs to publish benchmarks, flowery advertising or not. But solid benchmarks are incredibly difficult to do. Have you ever published any? It takes enormous effort. Yet, I would excuse the words "blazingly fast" if at least some ad hoc benchmarks were run and at least some people are able to reproduce them, at least in common cases.
I think my bottom line here is that you seem to be complaining about a pattern, but I'm not entirely sure it is warranted.
I'm not sure citing uutils/coreutils as an example is fair. I love that project. I learned Rust contributing to that project, however, as the blog entry you cite itself notes:
> I saw the maintainers themselves mention that a lot of the code quality isn’t great since a lot of contributions are from people who are very new to Rust
I'm sure plenty of my slow code is still in `ls` and `sort` and that's okay?
It doesn't place them there, they exist there (due to inlining).
Rust libraries are designed to be fast when optimized with LLVM. Rust has a lot of layers of abstractions, and they're zero cost only when everything is fully optimized. If you look at unoptimized miri execution, or insert code-level instrumentation that gets in the way of the optimizer, these aren't zero cost any more, and overheads add up where they normally don't exist.
> Important note: Miri is not intended to accurately replicate optimized Rust runtime code. Optimizing for Miri can sometimes make your real code slower, and vice versa. It’s a helpful tool to guide your optimization, but you should always benchmark your changes with release builds, not with Miri.
Sweet tip.
I'm not sure if it exports results in a format Chrome can render but it does produce great interactive SVGs and is compatible with speedscope.app
these two articles also showcase how to use chrome for this
Yeah that's my general problem with all these flamegraphs and other time based tools. There's a bunch of noise!
I'd image for something with deterministic GC (or hell no-gc) you should be able to get a "instruction count" based approach that'd be much more deterministic as to what version of the code is fastest (for that workflow).
The bottleneck of modern hardware (generally) is memory. You can get huge speedups by tweaking the way your program structures and operates on data to make it more cache friendly. This won't really affect instruction count but could make your program run 2x faster.
The latest iteration of this thought process was wondering what would happen if you exposed microcode for memory management to the kernel, and it ran a moral equivalent of eBNF directly in a beefed up MMU. Legacy code and code that doesn't deign to do its own management would elect to use routines that maintain the cache abstraction. The kernel could also segment the caches for different processes, reducing the surface area for cache-related bugs.
[1]: https://web.stanford.edu/class/archive/cs/cs107/cs107.1202/r...
[2]: http://www.codeofview.com/fix-rs/2017/01/24/how-to-optimize-...
A long time ago when people still tried to charge for profilers, I remember one whose primary display was a DAG, not unlike the way some microservices are visualized today. Only the simplest cases of cause and effect can be adequately displayed as a flamegraph. For anything else it's, as you say, all noise.
The value of the flamegraph over the previous iteration was that it more directly showed the chain of events. So no, it doesn't measure it 'just fine'. It's accomplishing absolutely nothing.
Yes, and it continues to show that.
A perf trace for:
async fn foo() -> i32 {
bar().await
}
async fn bar() -> i32 {
baz().await;
whatever()
}
#1 whatever() (somewhere inside it)
#2 bar::poll() (at the cpu_intensive_i32 call)
#3 foo::poll() (at the bar.await line)
#4 runtime executor ... (internals we don't care about)
>So no, it doesn't measure it 'just fine'. It's accomplishing absolutely nothing.
How about you actually try to use it instead of assuming?
That’s a toy example and if you can’t see the bottleneck in a toy example then good luck to you.
If you’re trying to figure out why 5 service calls followed by two more to merge the data together is taking 70ms, it’s not going to be very helpful. If you’re trying to speed up a batch process that makes 400 calls it’s only going to tel you about the cold start and the long tail. Both important problems but not your biggest issue.
Would it be a reasonable use of resources to run all those test suits and identify hot spots for community wide optimization?
Yes, that "massive list" being every single crate in the crates.io repository.
> Would it be a reasonable use of resources to run all those test suits and identify hot spots for community wide optimization?
I believe the approach on the perf side of things has been to take reports on crates that are particularly slow (even at one particular part of the compiler) and create benchmarks from those. I believe this is partly because running against every single crate would be too slow, and partly because it would be a moving target (as new versions of crates are released) and thus would make it hard to track performance accurately.
For example you could imagine noticing that people make a lot of Vecs with 400 or 800 things in them, but not so many with 500 or 1000 things in them and so maybe the Vec growth rule needs tweaking to better accommodate that.
Any front-end devs reading this? :)
Edit: Maybe something like this: https://github.com/guillon/run-qemu-profile