The Downside of the common man having too little oversight is more people believe things that aren't true (Flat Earth), and the downside to too much oversite is people believe different things that aren't true (PRISM, Nayirah testimony, Tuskegee Syphilis Study). The trade off of which is worse varies, but generally most individuals are fairly powerless but there are many of them, and organizations are few but wield much more influence.
They're bad we get it, Id rather be able to hear how bad they are in the light. Who was accountable when we spread lies about WMDs in Iraq? That caused magnitudes of more damage than these groups, yet no one is held accountable even with Public figures. Do you want everyone accountable for misinformation silenced or just those you disagree with?