If you do it with multi-stage build (where you would copy cached packages in the intermediate stage and remove it after being done) and copy over the whole root (/) from intermediate to the final stage then docker will make a new layer with the completely the same content and your image size will double -> this is a no go.

And yes, you can spin up an http server and serve the cached packages and setup up all this before using docker build, but hey I can do it with docker run + docker commit much quicker... (I wrote "you almost can not create" and not "you can not create" - so it is possible with Dockerfile, but is not trivial.)