Hmmm, I doubt that taking an OS project course or reading some virtualization research papers gives anyone the amount of insight into the nitty-gritty details needed to make a production-quality VM product. There is simply no substitute for thousands of hours of hard work pushing the boundaries on a field in order to get that required expertise.

Yes, perhaps today someone could improve upon VMware's products and form a competitor, but when VMware was first forming, I doubt that anyone else had the required expertise besides the few research groups who were actively working on virtualization for PC-class machines.

One alternative option is to be a founder who raises money and then hires someone who has actually done that research.

(Of course, I'm only talking about startups founded on research technologies ... you don't need a Ph.D. to found the majority of software startups)