Rebuilding by pulling in all your dependencies from the Internet again? That's not something I want to do when the clock is ticking to get a security update out. What if some random third party dependency is hosted somewhere that is down, or has been pulled?

And how do I know I have to rebuild in the first place?

You'll probably tell me that there are tools to help deal with this. I'm sure there are. My point is that it isn't as simple as "scp".

Or doing patches. Copy the file locally... apply the patch... restart process.