Imagine a Windows based on Unix, like OSX, instead of being a derivative of VMS. It could so easily have happened.
The problem was the absolutely awful userspace and Win32 they pasted over it. And then let that dig tendrils down into the lower levels of the kernel.
I remember administrating systems running NT 3.51 and finding it actually not that bad apart from the usual Windows fault of having to use the GUI for almost everything.
I doubt that it would happen given that Microsoft's modus operandi is to make as much money as possible by trying to lock people into their products, with Windows being the primary product on which they want people to be dependent.
https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process...
Drawbridge powers the pico kernels infrastructure on Windows and is the basis of the new Ubuntu/Windows subsystem.
The research in Singularity and Midori were used in MDIL compilers targeting .NET AOT native code generation on Windows 8.x and the follow up changes that lead to .NET Native.
Also F# started as a MS Research project.
The theorem provers used to validate Windows drivers were also a MS Research project.
LINQ, .NET contracts and Fakes framework also started as MS Research projects.
Some Haskell and OCaml researchers are on MS Research payroll.
There are tons of other examples.
How easy is it to port the average program from MacOS to Linux?
While the Kernel may be close enough, the window library is so vastly different that it's probably the same work to port from windows to Linux as to port from MacOS to Linux
The other issue was performance. UNIX systems at the time needed a far faster CPU and graphics, and far more RAM and disk space. Faster = much more expensive. Which is also why Windows systems were 1/4 the price.
Xenix collector here, I would say they had (sort-of) MS-DOS binary compatibility, as you could run some DOS programs on Xenix 386. I think it's only limited to real mode programs, but I tried it a couple of years ago and it worked.
Not to mention that on Xenix you could develop for both platforms (well, more than that: with the right setup you could target DOS, Xenix 8086, Xenix 286 or Xenix 386), that's quite an advantage.
On a side note, a good collection of Xenix manuals can be found at http://www.tenox.net/docs/.
Sometimes, it even worked!