"There is no reason for any individual to have a computer in his home"
"Unix is snake oil"
—Ken Olsen (DEC Founder, president, and chairman)
Also prior to that OSs were proprietary - if you didn't work for a hardware manufacturer you mostly didn't get to play with them much. There was no incentive to make them portable - then along came Unix and it was dropped into the hands of a generation of college kids.
Sure VMS had a better VM story (I both ported Unix and was a VMS systems hack at the time) - but you couldn't pick it up and put on anything other than a vax, it was pretty useless as an OS unless you were DEC, Unix didn't have that yoke around its neck (it had others mostly because AT&T didn't understand what it had)
I can go on and on ...
Unix "won" by becoming the darling of the people not willing to pay money while Windows swept the field of those willing to cough up cash.
However, don't mistake the virality of Unix for it being "better" than VMS.
The thing is you're comparing 1 VMS that didn't evolve much (it did get sort of cloned by MS) with effectively hundreds of Unix descendants.
As I said I've worked intimately with both, and I don't think VMS gets as much love as it's due, but in the long run I think it was a dead end largely because it was held close by a (great!) company that died
I think this same system oriented view (as distinct from user) is what sunk DEC, HP and Sun.
They were aware of what they had, hence why their first action after being allowed to commercially take advantage of UNIX was to sue BSD.
The lesson I take from tech of this era is that good enough (and cheap) beats superior over time. Unix over VMS, Ethernet over Token Ring, etc. I posit that Unix's simplicity of that era (you really could hold the entirety of the system in your head back then, and rereading the entire manual set every few months was something considered perfectly feasible and regularly recommended for power users) was the winning formula. It certainly helped that unix was all over academia. 20 years later, when Solaris was the incredibly powerful big dog and linux was the upstart, history repeated itself.
VMS wasn't all vax assembly either. VMS and a lot of VAX system software was mostly written in BLISS [1], a high level language intended for systems programming that was created about the same time as C. It was all there for customers to read in a big stack of microfiche cards delivered with each machine. I remember being shown the code for the VMS scheduler or maybe the dispatcher which was 'a page or two of (VAX) assembly' - I recall it was pointed out that it was one of the (few) places in VMS that did use assembly.
I don't know how portable the BLISS code for VMS was, but there were BLISS compilers for several other machines besides the VAX.
A similar problem (AFAIK) befell BLISS - BLISS code for the VAX treats all values as 32-bit. There is a separate BLISS-64 compiler for the 64-bit platforms VMS was ported to, but I don't think all the BLISS-32 code was converted over to BLISS-64.[2]
[1] http://www0.mi.infn.it/~calcolo/OpenVMS/ssb71/6467/6467p005.... [2] https://www.cs.tufts.edu/~nr/cs257/archive/ronald-brender/bl...
So the question is why were all these university people running Unix instead of VMS. Were they just dumb?
The "university" people were still running VMS certainly up through 1990 or so. Very few people ran Ultrix on their VAX hardware. And workstations were never widespread in universities as workstations basically cost as much as a car. A lab with 8 workstations was still a big deal in 1990.
Unix didn't sweep away VMS, PCs did. Just like PCs swept away workstations.
Unix didn't "win" until it latched onto PCs as a host. And people also forget that Unix didn't "win"--it lost terribly to both Microsoft and Apple.
It seems, no, DEC did not miss the PC boat due to one person's nearsighted view.
Microvax was 1985 and yes it would have been interesting if they had a low-end personal product based on VAX at that time. They did have the Professional [0] around 1982 based on PDP-11 which I used at school to learn Macro-11 assembly, C and Pascal. It was blazing fast and ran RSX/11 which was years ahead of DOS. They also had the Rainbow [1] around 1982 (maybe marketed to officework) with a z80 and 8088. And then there was the VAXMate in 86 which we don't talk about :). So it seems these were all aiming at the new "PC" market in a scattergun manner but nobody really knew what that was at the time.
0. https://en.wikipedia.org/wiki/DEC_Professional_(computer)
Between non of them being a actual PC clone, not running the same software and not really able to compete in the race to the bottom price fight they simply failed to get traction.