No, you also have a right to free speech against private actors (in certain circumstances). For one example, see Snyder v. Phelps.
It is true that there are exceptions to the "congress shall make no laws ..." part of the first amendment; so by analogy, there might be some exceptions to the GPL's "no further restrictions" clause, but you have not explained why grsecurity falls into any of those hypothetical exceptions.