Doubly so since when installed on a server very often it will be done as "sudo pip install ....." (/s/pip/other-package-manager/ as needed)
Running "pip install" as a user that has access only to the virtualenv directory is sounding like a good strategy.