Who gets to make the distinction, or decide what Free Speech is really "about?"

If the answer is anyone but the people engaging in the speech, isn't that a little dangerous?