It’s a bit of a chicken and egg problem that we can really only fix by allowing content to explicitly opt-in to linear working color profiles. And sadly that’s not going to solve everything either: while a linear profile is good for color interpolation, sRGB encoding is ideal for brightness interpolation. You’re stuck choosing between ugly red-green interpolation and nice white-black (sRGB) or between nice red-green and ugly white-black (linear). There are some more complex color spaces that do try to be the best of everything here, but I’m not aware of any browser or OS providing support for working with these perceptual color spaces.
A similar problem comes up when trying to manage color gamut on screens and systems that are intentionally blown out to look vivid, vibrant, shiny… looking at you Samsung phones. Any properly color-managed image there looks incredibly dull compared to the bright defaults you’ve grown accustomed to. If we take 0xff red to mean “full sRGB red”, that’s much less bright these days than if we just assume it means “as red as the screen will go.” It’s much like how TVs are (hopefully were?) set in the store in bright shiny mode to induce you to buy them, not set to reproduce colors accurately.