Estimated Cost to Store All US Phone Calls Made in a Year
docs.google.com
docs.google.com
A number foreign blogs/companies/people have displayed outrage that, given that they're not protected by the legal framework American citizens are, PRISM may give the NSA access to their data.
Irrespective of the truth associated with this, you've got to expect at least some percentage of paying customers will move business away from American companies. Considering this, is it conceivable that companies like Apple, Facebook, etc, could sue the government for lost earnings as a result of the fallout from this? Or are there a bunch of reasons why they wouldn't/couldn't (other than the obvious ones like, don't piss off the government).
Where would the traffic go?
It's hard enough replacing products & services from the 'big companies' (google, facebook apple, etc) with smaller ones operating inside the US. Let alone replacing them with services outside the US.
Where does the traffic come from?
Do the majority of users really care that they're being spied on. "If you have nothing to hide..." seems to be a reasonably common way of thinking.
That's just one example, and a friendly one. The not-so-friendly example is Qwest and Joe Nacchio. Or perhaps you lose lucrative government contracts for iPads. Or perhaps the Senate massively turns up the heat on your tax avoidance schemes, and it costs you billions because they have a record of everything you've ever said or done via NSA spying and know where you broke some obscure tax laws (guaranteed to happen at that scale). Or perhaps that damn pesky anti-trust case just refuses to go away, and instead is expanded, as the Feds start snooping around other business deals and practices (guaranteed to find something).
Or if this were the past, maybe they light Steve Jobs on fire in the options scandal, instead of being nice and letting it go away with the equivalent of a slap on the wrist (with a few patsy scapegoats).
Politics is the art of compromise.
It's not about who is right and the right thing happening. It's about who can best navigate the system to achieve what they want, or, as much as they can without losing to much.
Your comment covers this concept perfectly.
The NSA has a $15+ billion budget. The FBI has a $8 billion budget. The US military has a $638 whatever odd billion budget. The intelligence budget is $80 or so billion.
Yeah they can afford it. That is not an issue.
iow Many fortune 500 companies and foreign governments, all with less legal scruples, security and obligations than the NSA could also store every single phone call made in the U.S. - if they can get their hands on a copy.
Hell, I imagine any decent organized crime syndicate could scrape up $27M to store all the data and start mining it for information about when homes are unprotected.
Having all of these calls stored in one place is a HUGE liability. I said -if they can get their hands on a copy- which will be hard to get it from the NSA... unless there are inside jobs at the NSA and unless there are no external contractors with less scruples, security, etc. in place.
IOW, I'm not as scared of the government having access to this data (although I'm against it), but it's even scarier that 3rd parties can gain access to it.
I'm assuming in my example that the government isn't the one giving the calls / data to a private company, and that said company is doing the spying itself.
It was illegal for the telcos to go along with the first round of warrantless wiretapping, it took an act of congress to make it retroactively legal.
But I believe there's value in keeping the recording, for voice matching, recording of other sounds in the call, etc
The base here is developed from the author's "family average". That doesn't, in any way, reflect "all US phonecalls". Consider business users. There are a substantial number of business users who talk on the phone for >1,000 minutes per month. "Family" averages are only going to reflect personal phone calls, which are a fraction of the phone calls made.
We also cannot assume equivalency between what the Internet Archive pays per petabyte and what the NSA pays per petabyte. When dealing with government projects, you have all manner of requirements that have no parallel in the rest of the business world.
That $27.2 million number might as well be $50 million, or $100 million. It all depends on your input variables. This is napkin math at its worst.
Being able to quickly do order of magnitude estimates (especially in your head) is a very important skill IMO. I've heard McKinsey makes it part of their interview process.
Let's compute an outer limit: record everyone, all the time, CD quality.
44100 samples per second * 2 bytes per sample * 2 channels * 60 seconds per minute * 60 minutes per hour * 24 hours per day * 365 days per year * 313900000 people * $100 per terabyte / 1 terabyte = $175 billion per year. That's an absolute outer limit for cost.
That's less than 5% of federal budget, and we haven't started on the obvious ways to cut costs by several orders of magnitude. Reduce it to 4410 samples/sec, 1 byte per sample, 1 channel, 1/10th the time and we're already under $0.5B/yr, without even addressing audio compression (much less voice-to-text).
Telephone codecs are extremely low bitrate (relative to something like music, much less video) because you have very clear design constraints, and those constraints are forgiving. You just need to be able to understand the caller's voice, not accurately reproduce a live performance of Beethoven's 5th. I agree that the estimates to store the data are on the very, very high side, but I'm not sure that number is even significant in the whole scope of the challenge.
The challenges in obtaining every phone call made in the US aren't storage related; they're almost entirely collection and aggregation related. These ballpark numbers don't even attempt to factor in that portion of the cost. IMO, the collection, transport, and aggregation systems are easily 8/10ths of the problem, not storage. Storage is an extremely scalable solution.
If the government were going to do something like this, they'd likely tap in to the phone networks at the same places everyone else does. There are major telecom aggregation points - called "tandems" [1] - around the country. If you want to set up your own phone carrier with an actual physical network of your own, this is where you plug in. The government would have to do the same. They can't simply tap in at a single aggregation point, because not all phone calls pass through the same points.
From there, you face the choice of storing the data regionally in several data centers, or attempting to aggregate it all back to a central data center. IMO, a disaggregated approach makes a lot more sense. You could reasonably expect to transport all CDR [2] data back to a central location, but you wouldn't want to send all media (audio data) back to one place. It would be simple enough to only fetch what you need based on a query against call data. You'd want all the the CDR data in one place so you could perform your "big data" analysis on it efficiently, then cherry pick media to pull in for analysis.
I'm certain I haven't even scratched the surface here. This only gets the government long distance calls. It doesn't touch local, or even intraLATA calls. Maybe the government isn't interested in those calls though. Maybe they're only interested in international calls, which makes the problem simpler, not harder.
Others in the thread have mentioned that this should be treated as an "order of magnitude" estimation. I don't think we can draw any inference from this estimation at all, because it represents such a small portion of the problem domain. We also have no idea what the scope of the challenge is.
The entire exercise is pointless when you think about what we're asking. "Can the government actually implement a solution to record every phone call in the US?" I think that's a resounding yes. The solution would look a lot like setting up a tier 2 network provider [3] with extra investment in a storage back end. That's entirely within the realm of possibility given the size of the US Dept of Defense budget.
[1] http://en.wikipedia.org/wiki/Class_4_telephone_switch
In this case, I was responding to "You might as well throw out any number" by throwing out a ballpark figure for the absolute worst case audio surveillance cost scenario: everyone, all the time. OK, so the final number is really huge, and doesn't take into account the many nuances you mention...but I know that reducing storage costs by orders of magnitude will leave plenty of room to accommodate your real-world details. With that sweeping overgeneralization, I've concluded that whatever the details of implementation and whatever the extent of monitoring desired, the cost is well within the operating expenditure of the US federal government.
Now that we know that it's not realistically going to cost more than 5% of gov't spending, we can contrast that with the NSA's actual budget, and guesstimate how big the eavesdropping effort really is from that.
If nothing else, it's an exercise to explain to common readers (not you, you're a telecom guy who groks this stuff) that it IS in fact possible (not easy, not cheap, but indeed practically possible) to record every phone call. The scale of capability of some modern technologies is otherwise incomprehensible to most people; they'll dismiss the notion out of hand unless you can quantify it in very simple terms they can instantly grasp, like "recording everyone 24/7 would cost no more than 5% of federal operating costs."
Kinda hard to mentally keep up with technology which has expanded a billion-fold in capacity in just 3 decades.
That would give you an idea of the size we're talking. It's really not all that big. I don't think people realize that the technology that drives telephone networks (especially once you get outside the last mile) lends itself very well to snooping.
(Credit: commenter petrilli on Schneier's recent post about this.)
The only thing it does not take into account is business to business calling.
http://ivymike.blogspot.com/2006/02/could-nsa-wiretap-everyb...
I used a different methodology to come up with call volume; I looked at the Inter- and IntraLATA numbers (in 2004, which was the lastest available when I did this analysis.) My number was 700 billion minutes per year.
This new estimate is around 1100 billion minutes per year, which seems very plausible to me.
At 1.5M square feet, it could hold 344 copies of the national phone call audio database, based on the OP areal estimate.
An unconfirmed report [0] asserts the center will store 5e21 bytes. World internet traffic is 3e21 bytes in 2012
http://www.npr.org/2013/06/10/190160772/amid-data-controvers...
The estimated power of those computing resources in Utah is so massive it requires use of a little-known unit of storage space: the zettabyte. Cisco quantifies a zettabyte as the amount of data that would fill 250 billion DVDs.
"They would have plenty of space with five zettabytes to store at least something on the order of 100 years worth of the worldwide communications, phones and emails and stuff like that," Binney asserts, "and then have plenty of space left over to do any kind of parallel processing to try to break codes." reply
Exabyte storage seems possible, however.
A mere 500 2.7EB complexes and it starts looking like real data.
1M sq ft of total area, and 100k in the data center may not preclude another 100k sq ft of 'warehouse' containing millions of carts in shoe boxes with GS5s running around in sneakers.
Cheaper power = more cost-effective storage.
There's some expensive stuff in there.