I'm okay with that. The whole idea of our government and society is that the mass of law-abiding and decent people are stronger than the criminal and malicious minority. People are by and large responsible, which is why they can and should govern themselves. Limiting the government's ability to snoop and intrude on citizens is a crucial check on the very real (if long-term) threat of government over-reach.
But let's not kid ourselves that our privacy, and its constraints on the government, is without consequences.