Yes, you at least have to say where to get the source from if you publish/distribute binaries to others.
You do not have to distribute proprietary source code which has not been incorporated into the GPL code (the definition of this varies depending on whether the GPL code is GPL, LGPL, etc.)