This has been the #1 cause of quality issues for my (commercial) projects.
If you start a project with a new chipset, with a new vendor - build a new VM, install the vendors tools (and only the vendors tools) in that VM, and do your builds from there.
Do your hacking on your own (non-VM) machine, sure. That's perfectly fine.
But ALWAYS use the VM to do your releases. And, for the love of FNORD, KEEP YOUR VM IN SYNC WITH YOUR DEV WORKSTATION.
Disclaimer: currently going through the immense pains of dealing with a firmware build that absolutely has to be fixed while the original dev is on vacation - nobody can get into their workstation, the VM prepared for the task is 6 months out of date, and the customer is wondering why they should pay for the whole team to do something that one very special programmer is the only one on the planet can do .. grr ..