What would I suggest otherwise? Don't laugh, but something like NPAPI or ActiveX (COM). We had all these great component models in the 2000s. We could put them in a lightweight VM or a container for security. Instead of creating a window, we could let them directly draw to a shared-memory texture, which would solve some usability problems. I believe we could solve all problems that these technologies had nowadays.
Fundamentally, something like this belongs into the OS, not the browser (or any application). You should be able to use every codec and widget from other applications. We had that around 2000 with COM and VB6: Drag the media player widget from a pallette, drop it on your form, write a line of code to load a video.
Also, nobody gets to decide what codecs you are allowed to use or not for "political" or strategic reasons. Can't play something exotic? Just download the best-rated codec from your OSes app store (remember, which would run in a sandbox). Creating content and want to make sure everybody can see it without installing? Do what you do already today, and use the greatest common codec shipped will all OSes.