Funny, I had a very similar experience at a client of mine last month. They were using Apache Spark images and installing all kind of python libraries on top of them. The biggest contributors to image size were:

- miniconda (~2GB)

- a final RUN chown -R statement (~750GB)

We reduced the image size and relative Spark cluster considerably by playing around with dependencies in order to stick with plain pip and using COPY --chown.

I also recommend [dive](https://github.com/wagoodman/dive) analyse what contributes to each layer.