Often users only want QEMU, with a graphical interface on top. Libvirt is not just a graphical interface, but also an abstraction over multiple virtualization backends.

It tends to add complexity, while making more advanced usage of specific virtualization backends more difficult. Especially with its own configuration format based on XML.

Edit: Correction, libvirt is not the graphical interface, but entirely the virtualization abstraction. Virt-manager one of several interfaces.