If I understand correctly, the reason why public photography is considered protected speech is because we want citizens to be able to capture public events of interest. That is, protests, riots, cops using excessive force, or more happy events like celebrations, etc. When it comes to someone taking photos in those situations I have no problem with it.
However I really dislike the commercial street photographers as a concept. Basically instead of paying models to work for you, you go out to use random people as models and then sell your photos as art. That to me is exploitation. I understand that legally speaking it would be incredibly hard to separate the two types of street photography so it must all remain legal, but at the same time I strongly dislike the latter category, especially for the photogs who act like entitled assholes about it.