For example, the "canPlayType" API for whether a codec is supported returns "probably" or "maybe". So we sometimes need to hardcode which browsers support which new codecs.
Also, there are several bugs in past and present versions of decoder implementations, which are only discovered though manual testing, or by observing quality of service metrics split by browser version (Firefox does not handle bad audio packets as well as Chrome, for example). In IE and old Edge, the video readyState would always be "4" after playback beings, even during buffering, which is a blatant violation of the spec Microsoft refused to fix (as stated in their bugtracker).
Browsers like Safari have subtly different event orders from the HTMLVideoElement which requires a burdensome workaround that us working in video just like to keep to Safari instead of poisoning other implementations.
A final fun quirk is that not all browsers gave accurate HTTP timing information until recently (notably, Safari < 14) which make download timing for the purpose of determining bandwidth very inaccurate. This is why Twitch only recently supported low-latency playback for Safari. There is no other way to ask "do you support accurate timing" other than for an engineer to test it and hardcode an exception.