The most efficient workflow I've found for pip so far is something along the lines of:
pip-compile # resolve and pin the dependency tree
grep '==' requirements-pinned.txt | xargs -n1 pip wheel # build every wheel in parallel
pip install *.whl # install the pre-built wheels
It may not avoid all the slowness, but when nicely integrated into a build/CI system it can avoid suffering it more than once.