If someone really just wants a delay, it's easy to either (for programs running on normal OSs) call a sleep function, or (on tiny embedded systems) add an empty inline assembler statement that the compiler can't see through.
Neither is measuring delays of cached versus non-cached instructions. Yet it turns out to be very observable.
Practically, if you need to execute a specific sequence of machine instructions in order to prevent side-channel attacks, then you have to rely on assembler, compiler intrinsics and/or OS support. But that was true way before Spectre.
I've been many loops that turn into no-ops because all the functionality has been refactored out but this fact is hidden in function calls.
Sure, this should ideally be surfaced as a lint error, not a compiler optimization, but you cannot say that intentional delays are the "only" reason.
Also since processing time is variable, using that as a method should be extremely heavily discouraged/warned/require-opt-in
It would be of course nice if a warning was produced for that specific case: This whole loop was removed - is it really what you wanted, or is it a broken delay loop?