Hacker News Reader

Or, record all of it that you can into a daily or weekly cache, and then keep in indefinite (expensive) storage the things which are Statistically Interesting but outside our current budget/capabilities to store forever.

View on HN

Yes, good point. You don't have to store everything forever, you can have tiers of interestingness — capture everything, read it looking for certain patterns, store anything that matches those patterns forever, store stuff that matches [secondary, less important but still useful pattern] for a few years, and store all other calls for a few months or a year.

Phone calls are quite low quality audio, but I don't expect the NSA to be limited to consumer grade text-to-speech technology, so at least for calls in some languages, they could store the transcripts forever.

EDIT: Apart from processing power, another expensive problem with such a setup is memory to store the firehose temporarily.

EDIT 2: If you were wondering, 200TB/day would run at $7500 for 50 4TB external hard drives at $150 each, assuming you wanted to use a Backblaze-like setup. In a year, that's $2.7 million. (This doesn't account for redundancy.)

Reply on news.ycombinator.com