The first amendment to the US Constitution provides very strong protection for the right to free speech. Presumably that is the origin of "free as in speech".
Do you think that the free software movement would have adopted "free as in speech" if the US was hostile to free speech?
Certainly, but it would have started somewhere that wasn't hostile to free speech. The Netherlands 20 years from now, maybe, or Korea when the computer is invented 150 years in the future in the early stages of the Industrial Revolution.
Right, but what about "free as in beer"?
If I buy you a beer, that's free beer for you.
Which, unless you're my friend, prompts the questions "what else is in this beer" and "what's in it for you." I believe that to be a deliberate choice in the "free as in beer" analogy.
"Free-as-in-speech, not free-as-in-beer" is just expecting you the hearer to understand that while event advertisements might prominently feature the words "free beer" as an exciting statement that beer is being given away for zero cost to drive interest, zero cost is not the sense in which "free" is being used in "free software"