PyPI Proxying for Docker Builds

> The main advantage of 1. is that it’s a bit simpler. My main concern with it is that multiple builds could occur in parallel that might cause issues. We had seen instance when running multiple instances of pip simultaneously caused the cache to become corrupted.

I believe that is what the ‘sharing’ option for cache mounts solve? Per docker docs[0]:

> One of ‘shared’, ‘private’, or ‘locked’. Defaults to ‘shared’. A shared cache mount can be used concurrently by multiple writers. private creates a new mount if there are multiple writers. locked pauses the second writer until the first one releases the mount.

So, for pip, can’t you just do:

    RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
    pip install -r requirements.txt

Although I’m not sure how that works with different python versions… Does docker link the cache to the specific FROM image? Probably not. I suggest using the 'id' argument with the python version, maybe:

    RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-cache-py3-12 \
    pip install -r requirements.txt

Although, the proxy solution is certainly a valid alternative.

[0]: https://docs.docker.com/reference/dockerfile/#run---mounttyp...

In the old world a simple global http proxy would suffice to catch all such traffic, but I guess pip is also protecting my package installation behaviour against eavesdropping of foreign intelligence.