This also works with others package managers/distributions. Only with apt/apt-get you need some extra configuration "APT::Sandbox::User=root"

I was thinking about using Ubuntu, but in the end took void linux because it's the smallest glibc based distribution I could found. Python wheels don't work on musl based distributions