Hacker News Reader

Comment by LAMike | Hacker News Reader

I wonder how many of these hard drives are needed to completely scan the internet every month and create a "time capsule" drive that you can revisit and browse ten years later like a yearbook

View on HN

Stackoverflow is about 10GB: https://archive.org/details/stackexchange

English Wikipedia is about 15GB: https://en.wikipedia.org/wiki/Wikipedia:Database_download#En...

Reddit is about 1TB: https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...

Common Crawl (May/June 2020):

"It contains 2.75 billion web pages or 255 TiB of uncompressed content". About 50TB compressed.

https://commoncrawl.org/2020/06/may-june-2020-crawl-archive-...

I think 100TB would be plenty. We're not quite there yet but I wouldn't be surprised if you could get "The Internet" on one consumer drive in a few years. This would be a cool product.

Something tells me that 50TB doesnt include video media. There are dozens of chinese pirate streaming sites with more movies and TV than whole of netflix and all its competitors. Then there is YT.

Or pictures.

If you look at that reddit link it's about "every single comment" -- and that's about 1 TB. I'd bet that the pictures from one fairly active if niche sub would exceed that, e.g. r/battlecars or r/breaddit

A fuckload. There are 160M active domains. Say 2% of them are news sites with tens of thousands of active articles, another 2% is e-commerce sites and forums with also tens of thousands of pages, and the rest have like 50 pages on average. So you have something like 154M sites x 50 pages each x let's say 200KB per page that makes it 1.540 TB of data. And for the rest 4% let's assume 20.000 pages on average by 2MB each, that gives another 250.000 TB. And then you add sites like YouTube which could easily triple the above numbers. You'll end-up with a million TBs at least. That's like 100.000 HDDs.

The website that lists all possible Bitcoin private keys would take up all of those drives anyway

With video and other rich media ? You will need hundreds of thousands or even millions of these disks.

YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day - 180,000 disks a year) [1]

If YT is no more than say 1% of the public internet (Facebook is probably storing lot more , most of the data is not public though, Instagram is ) Then you are looking probably at 18 million disks a year without any backups and copies considered .

[1] it is assuming SD quality 220 mb per hour single format and one copy.

> YT alone adds 500hrs of video per min . That is at minimum 9000 TB / day (500 18TB disks/day ...) ... assuming SD quality 220 mb per hour single format and one copy.

Nitpick:

Each bigger resolution stacks on top of the previous one, so 8K60fps means storing 8K60 (webm+av1), alongside 4K60 HDR (webm), 4K60 (webm+av1), 4K (webm), 1440p60 HDR (webm), 1440p60 (webm+av1), 1440p (webm), 1080p60 HDR (webm), 1080p60 (webm+h264+av1), 1080p (webm+h264), 720p60 HDR (webm), 720p60 (webm+h264+av1), 720p (webm+h264), 480p60 HDR (webm), 480p (webm+h264+av1), 360p60 HDR (webm), 360p (webm+h264+av1), 240p60 HDR (webm), 240p (webm+h264+av1), 144p60 HDR (webm), and 144p (webm+h264+av1).

And as noted, a given resolution is often stored in multiple video formats, so there's that too.

The data needed to store a single video thus adds up quickly:

  $ youtube-dl -F 1La4QzGeaaQ | grep -o '[0-9\.]\+.iB' | sed 's/MiB/*(1024^2)/;s/GiB/*(1024^3)/;s/.*/(&)/' \
    | tr '\n' + | sed 's/\(.*\)+$/scale=3;print(\1)\/(1024^3),"G\n"\n/' | bc -ql
  8.423G

So for https://youtu.be/1La4QzGeaaQ (a random 8K video I found), storing 5 minutes 37 seconds requires about 8.5GB due to all the transcoding.

It's quite possible YouTube also keeps the original uploads around for a bit as well.

---

It incidentally proved so complicated to correctly type out the format list above manually it was actually easier to figure out the following regex:

  $ youtube-dl -F 1La4QzGeaaQ | sed -n '/video/{s/^.\{35\}//;s/ \+[0-9]\+k \+, \+/ (/;s/webm.*\|vp9.*/webm/;s/avc.*/h264/;s/av01.*/av1/;s/4320p/8K/;s/2160p/4K/;p}' \
    | sort -hr | sed ':1;H;N;${:2;s/\([^\(]\+\)(\([^\n]\+\)\n\(\1\)(/\1(\2+/;t2;s/\(\n\|$\)/), /g;s/, $//};b1'

(The above took probably twice as long to type out as the list it produced, but it was a lot more fun, and importantly verified its own correctness, which I was having a hard time doing manually. Plus, it's cool to make sed do new things - like token deduplication and newline coalescence in this case (remove the 2nd sed command to see what I mean).)

The proposed project entails constructing a global-scale shared supercomputer capable of hosting the entire Internet on one platform as an application

https://en.wikipedia.org/wiki/IBM_Kittyhawk

So, basically where AWS is headed.

I joke, but there's a grain truth there.

It's true, though. IBM misunderstood the implications of cloud computing at the time.

They clearly realized their mistake:

In 2013 researchers used the Kittyhawk project to demonstrate a novel high-performance cloud computing platform by merging a cloud computing environment with a supercomputer.

Wow... is IBM the Kodak of computing/hosting?

If you actually do this, just make sure your crawler doesn't get caught on some programmatic infinite webpage generator.

Like one of those calendar widgets.

I suppose one could use the current size of the Internet Archive as a yardstick for how much public-facing stuff there is. Of course in general terms this also depends on your definition of 'the internet' - consider databases and other such stores. Huge numbers.

Reply on news.ycombinator.com