Throwing exceptions for everything isn't a catch-all solution for writing better software—take a look at the Ariane rocket failure, which was caused by bounded numeric types throwing exceptions aggressively. Some modern programming styles tend to avoid enthusiastic non-local returns anyway in favor of things like monadic composition, such as Maybe in Haskell (which is more like NaN than it is like an exception).

The nice thing about NaN is that you can just do a calculation as normal, and check for NaN at the very end. This means you don't have to do an expensive test/branch after every arithmetic or other numeric operation. The hardware is much, much easier to design if you don't have to make it branch, and the code is much faster without the compiler inserting branches. People who work with floating point numbers every day care very deeply about performance.

If you don't care as much about performance, why not write your code in a language that does throw exceptions? Python, for example.

Those of us that do use numerics love NaN, love signed zeros, and can live with NaN ≠ NaN even though it's kind of dumb.