On the other hand, it's kind of crazy no one can make an OS except Windows, Apple and Google? Trillion dollar market and no one can compete.
Now an OS without application compatibility is kind of DoA unless there is a very compelling reason to switch. Add in hardware compatibility and it gets even worse.
Much bigger hill to climb then incorporating an existing browser engine into a custom spin of a browser. Even a browser engine from scratch would be smaller than a new bare metal OS.
The OS game is over. Desktop computing is becoming a professionals-only thing. We can talk about pros and cons of Windows, MacOS, and Linux, but it's a shrinking market without room for a fourth player.
Now let's make that OS talk to a graphics card--whoops, no Nvidia for you, peon!
An OS isn't a problem. Hardware support on an OS--that's a huge problem.
With how mature the personal computer market is, this is a very big hurdle.
And that’s not even covering the numerous hobby OSs out there like Haiku, SerenityOS, ReactOS, TempleOS, SkyOS.
Then you have experimental OSs like Singularity too. There’s numerous examples of them alone but I think you get my point. :)
The way this all gets worked around is that people come up with stuff like Docker or Flatpak that ship their own copies of as many dependencies as possible. The disadvantage is that now I can't just patch an OpenSSL vulnerability by updating the system's copy of OpenSSL, the way Windows can for all software built on SChannel.
If you're looking for binary stability and to ship your app as a file, ELF is extremely stable. If your app accesses files, accesses the network through sockets, and use stable libraries like SDL or GTK it will work fine as a regular binary and be easy to ship. People just don't want to write their apps in C, when the operating system is designed for that.
Many native apps like Blender, Firefox, etc ship portable Linux x64 and arm64 binaries as tar gz files. This works fine. You can also use flatpak if you want automatic cross platform updates but yes, the format is unfortunately bloated.
It's not that easy to ship a JavaScript app on other OSes either and electron apps abound there too.
Also, I recommend taking a gander at what the Linux build process/linking looks like for large apps that “just work” out of the box like Firefox or Chromium. There’s games they have to play just to get consistent glibc symbol versions, and basically anything graphics/GUI related has to do a bunch of `dlopen`s at runtime.
Flatpak and similar take a cop-out by bundling their own copies of glibc and other libraries, and then doing a bunch of hacks to get the system’s userspace graphics libraries to work inside the container.