Modifications you distribute have to be given back. You don't have to distribute the modifications though.
But that's ok. If upstream is somewhat active then it's just a pain to keep maintaining your in-house patches, compared to sending them upstream. So that's automatically a motivator. If upstream is not active, then it does not matter anyway.
If I host GPL software on a webserver and my users use that webserver, I don’t have to give them the source code for modified GPL programs. This is fairly common.