GPL breakers wouldn't be breaking GPL if they kept their own work. They break it when they distribute a compiled work that's only partly theirs, and mainly somebody else's, without sharing the source code for it.

You're more than welcome to build on GPL code and keep your own work. Just don't distribute it.