I think this is a really naive view. Attempting to expand the "fire" in a crowd limitation to many other speech areas is reckless. The government are just humans, and humans are conniving and motivated by politics. That road leads to an obvious end.
You list small examples of when free speech goes bad but I can assure you not having it goes bad in a bigger way for many more people.