Not only contrast, but also the gamut. If display is capable of 255 levels of color and you remap 255 to e.g. 128 to leave the rest for HDR you have only half of the possible colors available for SDR. Or how is this supposed to work?
Also, per later comments, this feature only kicks in when you're actually viewing SDR content, so there's no downside (even contrast) in everyday use.
But it seems like Apple might be gradually moving from 8-bit to 10-bit displays. They don't really advertise it clearly, but if the technical specifications say "millions of colors" it means 8-bit, if they say "billions of colors" it means 10-bit. (256^3 vs 1024^3.)
So if you've got the iMac model with the 4K screen, it's 10-bit and therefore capable of 1,024 levels of color. So remapping the UX to e.g. 512 will still be fine, assuming (safely, I think?) the compositing is all done using 10-bit color.