I don’t think your Dockerfile should be downloading your dependencies and building from scratch. Let your build pipeline pull them in, run tests, and pass them verbatim into your container.
Vendor your dependencies if you have to, or maintain a cache, but don’t make your Dockerfile redo all of that work.