Depending on what you are patching, it will invalidate a great part of the cache, and you will be looking at very long build times for everything in your system.
If it’s only dynamic linking than yeah, it might happen that you need a huge recompile (but that is not that big of a problem nowadays in my experience - gentoo used to compile way longer in my subjective experience for example). Also note: nix will soon get content-based hashing which may solve this problem.