Given qemu's security track record, they're not necessarily wrong.
Chances are what you want is "simply" access to a shared folder rather than root.
1. Most users won't be affected by all the exploits (you don't stuff in a VM all models of network cards, SCSI controllers, etc)
2. Many deployments of QEMU (through Xen or Libvirt) are protected by AppArmor/SELinux. This would at least forbid access to /proc/self/mem but I can't say if this is enough to prevent evasion. IMO, this is likely to make the task quite harder.