Throwing exceptions for everything isn't a catch-all solution for writing better software—take a look at the Ariane rocket failure, which was caused by bounded numeric types throwing exceptions aggressively. Some modern programming styles tend to avoid enthusiastic non-local returns anyway in favor of things like monadic composition, such as Maybe in Haskell (which is more like NaN than it is like an exception).

The nice thing about NaN is that you can just do a calculation as normal, and check for NaN at the very end. This means you don't have to do an expensive test/branch after every arithmetic or other numeric operation. The hardware is much, much easier to design if you don't have to make it branch, and the code is much faster without the compiler inserting branches. People who work with floating point numbers every day care very deeply about performance.

If you don't care as much about performance, why not write your code in a language that does throw exceptions? Python, for example.

Those of us that do use numerics love NaN, love signed zeros, and can live with NaN ≠ NaN even though it's kind of dumb.

Because having the hardware throw exceptions all over the place can get INCREDIBLY tedious if you want to ignore them and solider on. It means having "try/catch" around every single arithmetic operation everywhere. So it's tons of extra code and it might slow things down 100%-500% or more.

What if you're doing some kind of matrix math and you just need the results of the diagonal and a single NaN creeps in somewhere but doesn't destroy your matrix. Would you accept your program needing 50% more LOC and taking 4x as long for this result?