Hacker News Reader

Comment by XorNot | Hacker News Reader

Neither of those two places could be described as having a particularly high volume of calls though. Its an order of magnitude or more difference in sheer volume you'd be looking at.

View on HN

Well, what would it take to record the whole US? One version says that we make ~5 calls a day on our phones [1] and ownership hovers at 80-90% of the 18+ population (and probably a bunch younger). The US is 311 million, lets say 80% of 300 million, so 240 million people making 5 calls / day.

The average call length is 1.8 minutes right now [2]. So we've got 240 mil person * 5 calls / person-day * 365 days * 1.8 min / call. So, about 788 billion minutes of calls / year at a flow rate of about 2 billion calls / day.

At MP3 compression of 128kbps, vocal data takes about 0.94 MB / minute [3]. So, close to 688 petabytes of storage data at a rate of about 1.9 petabytes / day. Seems within the realm of doable.

The problem of analysing this is in the "ridiculous parallelism" category, so they'd just be constrained by server farm capacity. Lets say they had a system with a conservative million nodes. Each day, each node would have to process ~2 GB of audio data looking for patterns. Not even challenging. If I were clever, I'd probably run a brute force audio to text on each node, then a text to symbol pattern analyser. I'd also have higher level net processes that look for patterns in calls spatially and temporally, but with far less processors.

[1] http://www.pewinternet.org/2010/09/02/cell-phones-and-americ...

[2] http://www.statista.com/statistics/185828/average-local-mobi...

[3] http://iaudiophile.net/forums/archive/index.php/t-2081.html

You can compress human speech while maintaining intelligibility much tighter than 128 kbps.

And did you seriously make a link to cite that 128 kb * 60 = 0.94 MB?

No, he made a link to cite that 128 kb * 60 / 8 = 0.94 MB

128 kb = 16 kB.

Your compression numbers are way off. 128kbps is for stereo music, not monaural voice recordings. If you stick with MP3, you can go down to 32 or 24 kbps without sacrificing any intelligibility. If you use a real voice codec like G.729, you can easily get by with 8 kbps. https://en.wikipedia.org/wiki/G.729

Here, have a data point: GCHQ had a sub-1kbps voice codec in the mid-80s (it sounds horribly 'squelchy'), and they used tape when they were doing it back then...

This is cool. Everyone always talks about the idea of recording and analyzing calls, but I've never thought about actual implementation. I appreciate the perspective.

Standard POTS audio is a single 3.4kHz bandwidth channel. Compressing VOIP codecs allow a wider range, but still sound great at 12kb/s.

So without any non-COTS tech and without sacrificing the lilt of grandma's sparkly voice, we're already at a 90% reduction of your numbers.

So let's go with 190 TB/day. Say we keep 30 days in hot disk storage, and spool it off to a digital tape robot afterward. We'd need a few thousand COTS hard drives on less than a thousand servers (triple it for decent RAID).

Then a few million-dollar LTO tape libraries, and a couple of guys to schlep tape cartridges all day long.

(Of course, since we don't require all that compute power, and because Gov't doesn't build datacenters like Google does, they'd just buy a few fat IBM z/OS boxes, and a few fat EMC cabinets for low $10's of millions total, and call it a day.)

Either way, it's well within the realm of eminently achievable, I'd say. Which means it's obviously happening. :)

Thanks for the correction on the compression that's possible (and others who offered comprehensible work in the 1 kbps to 30 kbps range). I didn't know the SOTA so I went for what I as a consumer thought of for audio compression.

On the final point, agree. This is in the range where its cost is round off error in some of the large security budgets. (+ probably far more for the labor, ops, ect...)

That was one of the main reasons I worked through it, because it sounds like a horrifically large task, but its really not even that crazy in terms of data rates and storage when you break it down.

And you can divide by two (person A calling to person B is just one conversation needing to be recorded, instead of two recordings for person A and person B), and you're at less than 100 TB per day. That's pocket money.

It's got to be two orders of magnitude easier for them to tap phones in the USA though. They don't have to ingratiate themselves with anyone or hide what they're doing, just show up at AT&T with a FISA-signed warrant and install equipment. https://en.wikipedia.org/wiki/Room_641A They've had a lot longer to do the work, and they have a lot less distance to transport equipment.

The call in Afghanistan is just satellite calls - those satellites are all US owned. Tapping them is trivial given the very limited number of satellites.

US telecommunications is geographically disperse, and my original point anyway was that the volume of calls is enormous - storing and processing that much data is hardly a trivial problem since it's both storage AND CPU intensive. From the US black ops budget you could make some reasonable estimates as to the total size and capability of the NSA to do this.

"The National Security Agency has been recording and storing nearly all the domestic (and international) phone calls from two or more target countries as of 2013." That seems like more than just satellite calls.

No. Mobile and landlines, too.

Satellite calls have been intercepted worldwide for decades now by numerous intelligence agencies.

And that's where the Utah data center comes in.

Reply on news.ycombinator.com