Even though they could do good, and may have in the past, the lack of actual accountability for their actions means that it's too easy to do bad. They're not directly elected, so even the tiny shred of accountability that elected officials generally have is not there. If they can flat out lie to the public and get away with it, why do they exist and why should they continue to exist?
The justification for the existence of a government organization should not be based on the possibilities of good they might do, but it should be based on the unchecked bad they could (and will eventually) do.