"Black Hole contains data collected by GCHQ as part of bulk “unselected” surveillance, meaning it is not focused on particular “selected” targets and instead includes troves of data indiscriminately swept up about ordinary people’s online activities. Between August 2007 and March 2009, GCHQ documents say that Black Hole was used to store more than 1.1 trillion “events” — a term the agency uses to refer to metadata records — with about 10 billion new entries added every day. "

I wonder if this "black hole" is backed by a huge HDFS cluster? When I was working in government consulting a few years back I used to alway see job posting at Fort Meade for Hadoop experts, and I know the government is (or was) a customer of Cloudera as of 2009. Incidentally, Amazon created a separate cloud service like EC2 for government data a few years back also I believe.

I assume the NSA must be utilizing Hadoop, HDFS, Impala and/or Facebook Presto - or have a system they built internally but never released that can processing and store data at the same order of magnitude.

If these documents are all true and we assume they are currently collecting more, not less data, then they were in 2012, then they are probably storing more data than Facebook/google on a daily basis

EDIT - i guess the might not be storing as much data as Facebook or google because this sounds like mostly text, no images or video. It still must be a shitload of data though! I also realize that GCHQ is not the same thing as the NSA, but I assume they are doing similar things with similar size data sets

Thoughts?