Hacker News Reader

Hacker News

Comment by VikingCoder | Hacker News Reader

Full thread

VikingCoder

Math is fun!

You can buy a 2 TB HD for $99.99 right now.

The population of the United States is 311,591,917.

That's 7057 bytes per citizen, for a penny under a Franklin.

The Complete Works of William Shakespeare can be downloaded from Project Gutenberg in EPUB format, in 2.3 MB.

Therefore, if you wanted to have William Shakespeare write as much text about every single citizen in the United States, as he wrote in all of his Complete Works, you would need 341.74 of those HDs.

Therefore, the raw storage would cost you $34,196.58.

The NSA's annual budget is around $3.6 billion.

Every year, the NSA could buy enough hard drive space to store 105,273 times as much text as the Complete Works of William Shakespeare, on every single U.S. citizen.

View on HN

Storing the text is easy - deciding what text is relevant enough to store is hard.

If you had the NSA's budget, you probably wouldn't need to decide what text is relevant to store.

Deciding what to store is highly correlated with what information are we most likely going to find useful.

When I finally bought an iPod, it had 60 GB of storage. I thought, "Which albums of mine should I upload to it?" Then I remembered, "Oh right - all of them!"

My point is that deciding what to store is expensive and error-prone, with false-positives and false-negatives. Storage is so cheap, that there's an argument to be made that they can just store everything they want to.

While you make a good point, your end number is wildly inaccurate.

1) I highly doubt the NSA is buying $100 dollar low grade hard drives. I bet this data is backed up at least twice per site, at multiple sites.

2) The data is not unstructured, it will be structured in some sort of database. Storing your facebook photos alone will be potentially hundreds of megabytes.

3) I doubt NSA is spending it's entire budget every year on hard drives.

Yes, that's why I quite carefully said, "Every year, the NSA COULD..."

Emphasis on the COULD. It's to emphasize how cheap storage is, and how relatively few U.S. Citizens there are. In our day-to-day lives, we think 311 Million is an enormous number. But in the scale of computing, it's actually quite small.

My real point is: storing the data is no longer a challenge. If you want someone (businesses, governments, etc.) to not store data about every citizen of the U.S., you need laws, regulations, and checks and balances.

1) If he's three or four orders of magnitude off, his point still stands.

2) Also, the data could be almost completely unstructured. You mine the data that you have to create structure. If you toss anything, or prematurely optimize things, you're missing out on what the algorithms that you come up with 10 years from now will give you.

3) If you're using enough redundancy to multiply $35,000 into $3,000,000,000, you're insane.

Why does this matter?

Reply on news.ycombinator.com