1. Getting enough "right" people who have enough time on there hands and care enough together to work on it

2. Lots and lots and lots and lots of testing in many different environments to make sure the replacement code is at least as stable and correct as the closed stuff

3. The possibility of patent and other IP related problems

3 may not be an issue (other virtualisation options like KVM and Xen have not been attacked this way (yet)). 1 is an issue right now but would probably stop being so if Oracle starting being a problem in this area (in which case the right people would be affected and either want to work on this for their own needs or would just jump ship to another solution). 2 would be the biggie though. Given how core the code is it would take quite some effort to prove to its potential users (including the people working on it!) that it is at least as safe as that which it replaces.

The one good thing Oracle did was put VirtualBox back into one version and make the extensions a separate package. Also adding usb 1.0 support into the main package.