I think that you've been lucky, since even for mostly-pure Python packages, you often can't install in one env and just use it in another, because there might be some deeper dependencies.

My wish would be for there to be a way to get "most of the installation" working in a portable way and for OS-specific stuff to somehow be isolated into a final step, but in the world of Docker images that feels like a rare user story.