I don't think this is true. Look at Zig, they don't seem to have a problem removing a lot of C's undefined behaviour, while still being able to surpass C in speed in many cases.
> Simple example: accessing an array past it's end it's undefined behaviour. If you want to remove this, you need to add bounds checking
Zig does indeed do bounds checking. I think the way it can still compete with C is: - Zig should be better at propagating constants (Better module system, Link-Time Optimization by default, and I think avoiding undefined behaviour helps here too). Arrays/slices do often have constant bounds. - You can choose to build with or without bounds checks (--release-safe, --release-fast). This means you're more likely to discover out of bounds problems during debugging, since you'll always get errors in those cases. But you have the option to release a fast version.
Julia has another interesting solution to bounds checking, where you can mark a piece of code with @inbounds to declare that you assume array access is within bounds.
I think some undefined behaviour can also be detrimental to performance. If you pass two pointers to a function, and it's undefined whether they alias or not, there are optimisations you can't do.