They're not the only consideration, but they are a major consideration. The public's tolerance of any privacy infraction will depend strongly on whether that infraction prevents 1 death a year or 1,000,000.
The public's tolerance of any privacy infraction probably depends a lot more on how scary the stories of potential harms used to defend it are than the actual harms that it has prevented or would be likely to prevent in the future.
Yes, the easiest way to do something might be to violate the Constitution, but that doesn't mean it's the only way.
Let's take the worst-case disaster scenario: Terrorists are going to get a nuclear weapon and blow up a major city every year. Millions of deaths per year.
Here are two ways to prevent this:
A) We read everyone's private correspondence without prior suspicion and arrest anyone who we think has obtained nuclear materials necessary to make a bomb. Obviously if we hadn't arrested them they would have killed millions of people, so reading everyone's private correspondence without prior suspicion has saved millions of lives per year.
B) We track down and secure all the nuclear material (e.g. what the Russians have lost) using traditional investigative work so that terrorists can't get it. Since it's too hard for them to make themselves, this prevents them from obtaining a nuclear weapon, problem solved.
C) We send undercover operatives into terrorist networks and disrupt them before they ever get off the ground.
You don't need A if B works. You don't need A if C works. You don't need A if any of D through Z works either.
1) Conserve energy through efficiency.
2) Develop renewable energy technologies.
3) Invent cold fusion.
Sure, we might not need 1) or 2) if 3) works, but without the existence of 3), it's very easy to make a case for 1) and 2).
And of course by the time that runs out, we'll have something like Planetary Resources up and running...
So while 3) might not ever happen, that does not imply 1) or 2) are the only remaining options. Proof by elimination only works if you can prove for all options, not just your preferred subset.