You're not sacrificing any control. I'll speak only of the ones I know: vundle, neobundle, and vim-plug are using very basic Vim mechanisms to save you the hassle. Appending to &runtimepath is the Vim equivalent of modifying your shell $PATH.
And deferred loading, if you choose to use it (I don't), actually gives you more control over plugin behavior.
> You own your vimfiles
This is emacs mentality. I don't want to own (read: maintain) the plugins I use. If I find a bug, I send a PR upstream so others can benefit. If the bug is unforgivable I uninstall the plugin. (You can also pin to a specific SHA).