You can resolve this by installing, building, and uninstalling in the same RUN command right? I once worked at a place that used some insane setup with images building other images to avoid this but I'm pretty sure the first option works as well.
I find that it's easier to debug the build process with a "standard" distribution.