This is a matter of choice, not something with an objectively correct answer. Every possible answer has trade offs. I think consistency with the underlying standard defining NaN probably has better tradeoffs in general, and more specific answers can always be built on top of that.

That said, I don’t think undefined in JS has the colloquial meaning you’re using here. The tradeoffs would be potentially much more confusing and error prone for that reason alone.

It might be more “correct” (logically; standard aside) to throw, as others suggest. But that would have considerable ergonomic tradeoffs that might make code implementing simple math incredibly hard to understand in practice.

A language with better error handling ergonomics overall might fare better though.