With virtualization you have the hypervisor that has applications running along with it (ie: ssh, a cli, syslog, bash etc) and then you install a guest in a VM on top of the hypervisor. The OS on the VM is another vector which has applications on it (DB, web, ftp, etc).
If I have a bare metal server with just an OS installed on it and its applications on top of that, I only have to worry about that set of OS and applications and their associated risk vectors.
If I have a bare metal server with a hypervisor and then the above OS and set of applications, I have increased the number of risk vectors by however many applications are running along with the hypervisor.
“but do people actually believe that it makes it less secure?”
Just bringing a hypervisor into an environment does not of itself immediately make it less secure. I agree with you, I don’t think it makes it less secure. It does increase the risk of the environment and appropriate architecture and action must be taken to prevent your statement from being true. A large number of environments do not architect and manage properly.
Another very realistic, and happening today, example: Bare metal server with Windows OS installed. Bare metal server with a hypervisor which just happens to have bash on it (or is susceptible to this memory issue). The same Windows OS is installed as a VM. In the second instance with the hypervisor I would have, indirectly and out of my immediate control, made my environment less secure.
An increase in complexity or increase in components will increase the risk of an environment.