Hacker News Reader

Comment by Dewie | Hacker News Reader

It's weird that people that make benchmarks don't investigate which flags to pass for getting the most optimized build. Many compilers don't do max optimization by default. In particular, people making benchmarks with rust code seem to tend to forget or be unaware of the `release` flag.

View on HN

Yeah, this is happening again and again in Rust: people publish Rust benchmarks without optimization on. In fact, we just decided this week to change "Compiling" to "Compiling (Debug)" in Cargo if optimization isn't turned on to address this problem. It's sad :(

If it helps it doesn't happen only to Rust but to all languages.

There is this misunderstanding between implementations and languages where people equate whatever they have installed on their computer with all implementations of the said language.

Also not knowing about profilers and optimization flags.

So yeah, it is sad.

Yeah, the performance benchmarks currently suck. They are neither statistically valid, comprehensive, and often just run with default settings. That was really just a personal notes file that I shared with def- without clarifying that it really wasn't ready for publishing.

That's probably because there is no --release flag:

    $ rustc --release prog.rs
    error: Unrecognized option: 'release'.

There is a -O for optimization, it is equivalent to -C opt-level=2.

EDIT: Oh, cargo build does have a --release which seems to be equivalent to -C opt-level=3, which I guess is even better.

`cargo build --release` does more than just `-C opt-level=3`, actually. For example a regular `cargo build` also adds `-g`, and that's removed for `--release`.

Maybe the release flag should be the default?

My preference is to have unoptimized by default, unless there is only a negligible difference in compile-time between the two options. I mostly don't need optimized programs while I'm developing them.

Alternatively, don't have a default and let people opt-in to whatever default they like. That forces people to actually make a choice, instead of being lazy and publishing poor benchmarks without having even looked up what optimization knobs there are to turn on or off.

Reply on news.ycombinator.com