How else could it be defended? Considering that someone has to show injury before courts will take up a case, it's usually going to be someone that's been charged with a crime whose privacy has been invaded.
Every time a case with an innocent starts winding through, the government drops the case, after having caused much grief to the innocent in question, rather than risk having their surveillance powers curtailed.