> ... I typically only have to pull the last few small layers because the base image hasn't changed.

That probably depends on your circumstances! For example, you could use a particular OS image as your base, software with updates as a set of intermediate layers and your software and whatever else you need as the last ones. That way, leaving the layers as they are would indeed result in some pretty good efficiency, since only the changed layers would need to be pulled.

Whereas if you base your software on a particular runtime image, e.g. OpenJDK or one of its varieties, then it's unlikely that you'll see such nice benefits, at least if you'll regularly update the version of the base image that you're using. Now, whether you should update everything that often in lieu of any serious security vulnerabilities, however, is another question.