...Is this running extra micro code to fix some hardware bug/unreliability? How can this happen? Doesn't look like a normal design decision.
Most "mundane" uses of floating point have no need for subnormal numbers, and results that underflow could just be flushed to zero. But they’re sometimes important in scientific computing to ensure sufficient smoothness around zero, avoiding precision issues.
If the use of subnormals is disabled with FTZ/DAZ that is guaranteed to generate big errors and it is completely unpredictable how big the errors will be.
If a computational algorithm generates underflows at some place, there is no way to modify the algorithm so that flushing-to-zero will not make any difference (i.e. no errors).
What is possible, is to modify the algorithm so that underflows will never happen.
This was the traditional way of writing numeric algorithms. Because on early computers underflows would crash the program, the same as overflows, one had to improve the algorithm in order to avoid both underflows and overflows.
Subnormals and infinities have been introduced in the standard precisely for lazier programmers, so that they would be able to avoid the rewriting of algorithms without the risks that underflows and overflows would generate major errors.
Unfortunately, it seems that for some programmers this is still not enough, because they want simultaneously to not be bothered with rewriting the algorithms and to have the program run as fast as with an optimized algorithm.
For this, the solution is very simple and it is not enabling FTZ/DAZ, which unless is done for a game might cause unpredictable financial losses for an unsuspecting customer, who expects that a computer must provide correct results.
The right solution is to not buy Intel CPUs or any other kind of processors whose vendor believes that the correctness of computations does not matter. It should be noted however, that the Intel server CPUs use CPU cores that are obsolete in desktop and laptop CPUs, i.e. the tested Intel CPUs use cores like those in Meteor Lake and Raptor Lake CPUs. I do not know if the more recent Intel CPU cores, from Panther Lake/Arrow Lake S/Arrow Lake H/Lunar Lake, have retained this Intel misfeature, which has characterized the Intel CPUs for much more than a decade.
If someone says that they have enabled FTZ/DAZ and they did not see any significant difference in the results of a program, that is complete B*S*T, because it is impossible to test exhaustively any program that does floating-point computations and the errors are expected to happen only for certain values, which are unlikely to be encountered during testing, but you cannot predict that those values will not be encountered in production.
More generally, subnormals are needed for Sterbenz Lemma to hold everywhere: https://en.wikipedia.org/wiki/Sterbenz_lemma
This was very good, because underflows completely break the assumptions about floating-point arithmetic on which numeric algorithms are based, so the errors in the final results become unpredictable.
Subnormal numbers have been introduced as a means to avoid handling every underflow exception, because typically the use of subnormals eliminates the errors that would otherwise be caused by underflows.
The flush-to-zero and denormals-of-zero options must be strictly forbidden for any general-purpose applications. They should be permitted only in applications where there is no doubt that regardless how big the errors will be they will not have any really harmful effect, which is true for games and perhaps for AI, but for little else.
This is another great misfeature promoted by Intel, in order to win meaningless benchmarks. It would have been much better if these standard-breaking features would not have existed, because they are much more often used when they should not be used, than when they are harmless.
Have a look at https://en.wikipedia.org/wiki/Subnormal_number for some context.
Some poster has linked a Mastodon thread, where Fabian Giesen explains that handling in hardware the subnormals is cheap in floating-point adders and in fused-multiply-add (FMA) execution units. Many processors do the multiplications in the FMA execution units, so there is no penalty for them to do the subnormal handling in the right way.
On the other hand, some CPUs, including the Intel big cores, have some floating-point multipliers that are separate from the FMA units. The reason is that those separate multipliers can have lower latencies, typically by 1 or 2 clock cycles, which may help those CPUs to win some benchmarks, especially when running unoptimized legacy programs (in optimized programs, most multiplications are combined with additions into FMA operations).
The separate multipliers are simplified in comparison with those included in the FMA units, and handling subnormals in them would be expensive, because then they would become so complex that there would be no advantage for them to be separate multipliers. Which is why Intel does not handle subnormal multiplication in hardware, but a microprogram is invoked for this.
I haven't done any work on this stuff since 2019, so my memory may be hazy.