Comment by araes | Hacker News Reader

Comment by araes | Hacker News Reader

Parent Full thread

Well, what would it take to record the whole US? One version says that we make ~5 calls a day on our phones [1] and ownership hovers at 80-90% of the 18+ population (and probably a bunch younger). The US is 311 million, lets say 80% of 300 million, so 240 million people making 5 calls / day.

The average call length is 1.8 minutes right now [2]. So we've got 240 mil person * 5 calls / person-day * 365 days * 1.8 min / call. So, about 788 billion minutes of calls / year at a flow rate of about 2 billion calls / day.

At MP3 compression of 128kbps, vocal data takes about 0.94 MB / minute [3]. So, close to 688 petabytes of storage data at a rate of about 1.9 petabytes / day. Seems within the realm of doable.

The problem of analysing this is in the "ridiculous parallelism" category, so they'd just be constrained by server farm capacity. Lets say they had a system with a conservative million nodes. Each day, each node would have to process ~2 GB of audio data looking for patterns. Not even challenging. If I were clever, I'd probably run a brute force audio to text on each node, then a text to symbol pattern analyser. I'd also have higher level net processes that look for patterns in calls spatially and temporally, but with far less processors.

[1] http://www.pewinternet.org/2010/09/02/cell-phones-and-americ...

[2] http://www.statista.com/statistics/185828/average-local-mobi...

[3] http://iaudiophile.net/forums/archive/index.php/t-2081.html

Standard POTS audio is a single 3.4kHz bandwidth channel. Compressing VOIP codecs allow a wider range, but still sound great at 12kb/s.

So without any non-COTS tech and without sacrificing the lilt of grandma's sparkly voice, we're already at a 90% reduction of your numbers.

So let's go with 190 TB/day. Say we keep 30 days in hot disk storage, and spool it off to a digital tape robot afterward. We'd need a few thousand COTS hard drives on less than a thousand servers (triple it for decent RAID).

Then a few million-dollar LTO tape libraries, and a couple of guys to schlep tape cartridges all day long.

(Of course, since we don't require all that compute power, and because Gov't doesn't build datacenters like Google does, they'd just buy a few fat IBM z/OS boxes, and a few fat EMC cabinets for low $10's of millions total, and call it a day.)

Either way, it's well within the realm of eminently achievable, I'd say. Which means it's obviously happening. :)

Thanks for the correction on the compression that's possible (and others who offered comprehensible work in the 1 kbps to 30 kbps range). I didn't know the SOTA so I went for what I as a consumer thought of for audio compression.

On the final point, agree. This is in the range where its cost is round off error in some of the large security budgets. (+ probably far more for the labor, ops, ect...)

That was one of the main reasons I worked through it, because it sounds like a horrifically large task, but its really not even that crazy in terms of data rates and storage when you break it down.

And you can divide by two (person A calling to person B is just one conversation needing to be recorded, instead of two recordings for person A and person B), and you're at less than 100 TB per day. That's pocket money.

Your compression numbers are way off. 128kbps is for stereo music, not monaural voice recordings. If you stick with MP3, you can go down to 32 or 24 kbps without sacrificing any intelligibility. If you use a real voice codec like G.729, you can easily get by with 8 kbps. https://en.wikipedia.org/wiki/G.729

Here, have a data point: GCHQ had a sub-1kbps voice codec in the mid-80s (it sounds horribly 'squelchy'), and they used tape when they were doing it back then...

You can compress human speech while maintaining intelligibility much tighter than 128 kbps.

And did you seriously make a link to cite that 128 kb * 60 = 0.94 MB?

No, he made a link to cite that 128 kb * 60 / 8 = 0.94 MB

128 kb = 16 kB.

This is cool. Everyone always talks about the idea of recording and analyzing calls, but I've never thought about actual implementation. I appreciate the perspective.

Reply on news.ycombinator.com