Actually that is not true. If you don't give someone the binary, you don't have to give the source.

So you can, if you want, sell the binary to select people (and ask them nicely not to give it to anyone else), and you only have to give the source to those select people, but not to anyone else.

The GPL does not require public source, only that the source follows the binary.