How a Theoretical Claim About the NSA Magically Transformed Into Factual Reality

This sort of statement is so misleading it hurts. It's kind of sad that people are so ignorant of what computers and modern data processing can do, that someone can go around saying this sort of things and not be laughed at.

The analysis of the citation chain is pertinent, though.

The actual headline he's decrying is 'Millions Of Gigabytes Collected Daily'. This is, according to the estimates and information he includes in the article, an accurate estimate. It is also suitably vague given the uncertainty over the estimates - it means anything 2 million gigabytes, up to 21 (the theoretical limit). We just don't know, what we do know is the ambition is to collect every communication, and they are nearing that ambition in reality, if only for a few full take days at GCHQ - the storage capacity in terms of internets/day at NSA I'm not sure on, does anyone know what their current capacity is?

Put another way, no! NSA and GCHQ are absolutely not gathering and/or analyzing that much data per day.

How does this follow? What has analysing got do do with collecting?

It’s an inconceivably big number meant to frighten readers.

2 petabytes per day would be just as frightening, and yet is well below the theoretical limit - I don't think the exact numbers here matter, this is an estimate meant to show just how much information is being collected.

But he saves the most bizarre assumptions for last:

As a group, and barring some sort of vortex, 550 analysts could only listen to 1,100 gigabytes of phone conversations per day, and that’s if they worked 24 hours per day and listened constantly. In reality, it would take 18,918 days or 51 years for 550 analysts to listen to just one day’s worth of fiber optic data gathered for Tempora.

Is he seriously contending most of this data is only telephone conversations, that collection without immediate analysis by a human being is not just as dangerous, that humans actually analyse each piece of information individually, without using filtering algorithms?

This comparison is far more misleading and inaccurate than the headline he decries.

He talks of the impossibility of the 550 (where has this number come from?) GHCQ and NSA analysts reading all this data. As if people are claiming that humans are individually watching it themselves. Even if they were, he completely ignores the obvious method: that machines under those agencies' control are the ones collecting this data, in favour of a hand-waving 'it's impossible'.

His statement that it's illegal for the NSA analysts to read these communications nicely sidesteps the strong evidence that they are doing it.

Even the focus of the article, the 21 million GB claim, tries to stoke up something much madder than what actually happened. The Guardian's reporting of the potential volume of data on the cables is not 'wild' in any sense. That the telephone game translated that potential figure into an actual one is not desirable, but also not particularly surprising. Notably, he doesn't take the opportunity to find out the actual average data rate on those cables and correct what he's criticising.

Of course, he also takes the opportunity to ladle on some criticism of Greenwald and the Guardian, and play up the 'it's a complex issue' angle which discourages any kind of public outcry.

Apache Accumulo:

Like Google, the [NSA] needed a way of storing and retrieving massive amounts of data across an army of servers, but it also needed extra tools for protecting all that data from prying eyes. They added “cell level” software controls that could separate various classifications of data, ensuring that each user could only access the information they were authorized to access.

http://www.wired.com/wiredenterprise/2012/07/nsa-accumulo-go...

Then:

How will graph applications adapt to Big Data at petabyte scale?

Brain scale. . .: 2.84 PB adjacency list, 2.84 PB edge list

Largest supercomputer installations do not have enough memory to process the Brain Graph (3 PB)!

[Accumulo gives] linear performance from 1 trillion to 70 trillion edges [on the 1 Pb Graph 500 benchmark]

From "An NSA Big Graph Experiment": http://www.pdl.cmu.edu/SDI/2013/slides/big_graph_nsa_rd_2013...

What would worry me is a chain of citations where the pertinent facts changed (e.g. "celebrity X has the potential for Y" -> "X did Y"). This happens too, as you sometimes find in issues of Private Eye.

In fact, by storing only what users send rather than what they receive you get everything that really matters.

This article and most others have failed to mention the technological force-multipliers (the bread and butter of HN) that their developers, sysadmins, and other technologist are constantly working on.

With the right search capabilities and filtering that 21 million GB gets a lot smaller.

always watching

This is an issue to be intellectually passionate about, not emotionally passionate. It's something I have to keep reminding myself of daily.

News outlets and bloggers are all too willing to play telephone and translate emotional energy into pageviews. That's not a good thing for the discussion.