O3 is a bad idea if the package is important. While that’s a dated opinion based on older experiences (Gentoo, various server software, etc), I suspect it’s still the easiest way to get exposed to compiler bugs.
The undefined behaviour is already in the C standard.
But you are right that enabling -O3 is generally going to make your compiler more aggressive about exploiting the undefined behaviour in the pursuit of speed.
In general, if you want to avoid being the guy to run into new and exciting bugs, you want to run with options that are as close as possible to what everyone else is using.
Even Design by Contract in Eiffel turns off assertions in production.
Rule #1 of programming: If it can go wrong, it will.