I make a point of never using pip outside of a virtual environment exactly for this reason. It has worked well!