I'll bet $20 your solution to the problems you posed is "Nix"
For a less controversial take, consider alpine's apk package manager. For a single-use container that runs one utility in an early dockerfile stage, apk can probably produce that image in 2-3 seconds, whereas for an apt-based container it's more like 30 seconds. That may not matter in the grand scheme of things or with layer caching or whatever, but sometimes it really does.