Oh cool, I actually hadn't seen pyenv before. Looks like it does indeed solve my problems (from a glance anyway, though I didn't see anything about pip in the readme).
It also has plugins to automatically work with venv, if you don't mind some 'magic' in your workflow.
Overall it's a solid setup.