You can avoid this by building cryptography as a docker stage and then just extract the wheel as part of your docker build process. Alternately if you install rust, build cryptography, and uninstall rust all in one layer then you can also avoid this issue.