Get used to this, watching people run benchmarks after having forgotten to compile with optimizations is basically a meme in the Rust community. :)
C has -O3, why isn't it the default? Because it takes a lot more time to compile. So I think no optimizations by default is the best choice. And I think Rust should do the same, you will be compiling more things in non-release mode than in release mode.
It may seem obvious to you and I that compilers have optimization levels, but consider how many people for whom Java, with no optimization level flags, is their only exposure to a manually-invoked compiler.