We're planning on something similar to move a big open source library somewhere else (so that dependencies don't break).

If you run pip install py.test it fails with a message that you should run pip install pytest.