Hacker News Reader

Comment by sydney6 | Hacker News Reader

No Worries! The Google Server in Mountain View has had a disk failed in it's RAID and therefore the sqlite-DB has only half the iops to it's disposal. A new 500GB hard disk is already in dispatch with Amazon. These are the moments why Google is a proud Amazon Prime customer.

View on HN

I know many people’s immediate reaction is that 500GB seems small, but remember, they’re not storing the whole internet — just an index of it.

The whole internet would need a much larger RAID, with literally dozens of hard disks.

They kind of do store the internet though, don't they? They store a cached version of most pages.

Tbh the source code of the whole internet would probably be a few PB at most, text is really cheap to store especially because it can be compressed. Images and videos are what makes the premise impossible because even with perfect compression you need an impossible amount of storage to store every video published by mankind.

Well, most of that is on youtube, so they kinda have a copy anyway :)

Should be noted HTML compresses extremely well.

∞ * .1 = ∞

But the 25TB you showed above is a better prospect.

Yeah but pages are HTML and HTML compresses extremely well. With the latest algorithms you could probably get as low as one byte per page. Probably even better with a decent middle-out compression algorithm.

(Also yes, you are correct within the realm of reality, but not within the realm of comedy.)

https://en.m.wikipedia.org/wiki/Yes,_and

Just take the “middle” out if the web. If it is more than 50kb excluding images but including css and scripts, just ignore that page.

To be fair, this is not actually all that far off the truth. Do napkin math on how much storage you'd need to index say 25 billion documents, and you'll see it's not actually a lot. 25 billion bytes is 25 Gb. How many bytes do you need per document (for the index)? A kilobyte, maybe? If so that's 25 Tb. While certainly not something you can fit on a thumbdrive, it's hardly something you need a planetscale computer cluster to store.

You may need it for the IOPS of actually using it, but that's another thing entirely.

Sometimes you read something so ridiculous that it feels true.

But i have to say: Personally, i don't run k8s and it's ecosystem and am just entering the whole "declarative world" with NixOS.. but nonetheless, these occasions are very rare, Google/YT Operations is just so impressive and has to be up there, if not a the very top.

He's probably more referring to the amazon prime thing.

> A new 500GB hard disk is already in dispatch with Amazon

What if in fact Amazon uses EMC gear underneath

Reply on news.ycombinator.com