This line of reasoning never made sense to me. Free speech is both a legal and a cultural issue. A modern society cannot credibly claim to be free on legal grounds alone. Yeah, the government won't put me in prison or fine me, but if I get fired for being a staunch liberal (or conservative) then I'm not really free, am I? This doesn't just apply to rich athletes -- in most organizations in SF I would not feel particularly safe expressing what seem to me fairly moderate views.
Depriving people of their livelihood for expressing political views is absolutely a free speech issue. Perhaps not in a legal/constitutional sense, but a cultural sense for sure. There were plenty of stretches in the Soviet Union where they wouldn't imprison you, but would fire you for a poorly timed political joke. Starting down that cultural slope is a really bad idea.