It's just as restrictive as any commercial license, except that instead of being prohibited from releasing the source, you are required to do so.
That is not 'free to do whatever you want'.
That is not 'free to do whatever you want'.
The GPL speaks for itself. You are free to modify GPL-licensed software and distribute your modified version with the caveat that you make the source code available. Thus you are in fact free to do whatever you want to the software. The restriction is on the distribution of the software, not the modification you make.
Why does that one restriction cause so much distress?