There is another Portable VirtualBox distribution[1] that normally does not install any service or driver. Only if you want to use a "Bridge Network" you need to install the drivers. I usually work with "NAT" option and in that case you don't need to install anything.

[1] http://portableappz.blogspot.com/2012/08/virtualbox-4120-801...