Hacker News Reader

Comment by bayindirh | Hacker News Reader

r/datahoarders will kindly and rightly disagree with you on that regard.

For an enthusiast, a 720TB array is pretty reachable. A dedicated enthusiast can get a 1PB flash array in 2U.

According to this article [1], 500 hours of video are uploaded to YouTube every minute. Depending on the video size and framerate, YouTube recommends up to 240 Mbps for 8k@60FPS [2]. Of course most video isn't that high res. Let's take a conservative guess that it averages somewhere between 2K and 4K and pick a middle bitrate of 24 Mbps. That's:

      24 Mbps / 8 bit/byte * 60 seconds/minute * 60 minute/hour
    = 10800 megabytes per hour of footage
    = 10.8 gigabytes per hour of footage

At 500 hours of footage per minute, that means 5.4 terabytes are uploaded every minute. Your 720 TB array would be completely full a little over two hours' worth of content that is uploaded to YouTube every single day, day after day.

At the current upload rate, 2,838.24 petabytes are uploaded every year.

I don't think you'll see hobbyist archives of YouTube any time soon.

[1]: https://www.tubefilter.com/2019/05/07/number-hours-video-upl...

[2]: https://support.google.com/youtube/answer/1722171?hl=en#zipp...

I wonder how much of the uploaded content is public versus private or unlisted.

For archival 24Mbps is an insane bitrate, you could get away with 1/3th to 1/6th of that.

You're also going to limit to public videos (unlisted and private will make up some share of those uploads) and probably to those with non-zero views.

I suspect archiving only videos with >100 views would probably cut the amount you archive to 1/10th.

This is a distinction without a difference, you’re still talking about O(1PB/day).

Some back-of-the-envelope math shows that YouTube would have to populate and rack a minimum of four 4U storage chassis (60 20TB drives each) per 8 hour shift to store that much. Roughly a little less than half a 42U rack. And that's before allowing for HDD drive parity, redundancy, and distribution across the globe.

1080p webm is around 450kbps and audio is 65kbps, so the estimate is of by 50x for the purposes of a hobbyist archive.

YouTube offers a number of codecs and nitrates. IIRC Opus goes up to 160kbps and m4a goes up to 128kbps, with lower bitrates also available. I imagine video is similar.

I don't think anybody on /r/datahoarders believes it's possible for a private individual to archive the entirety of Youtube. More than 200,000 hours of video are uploaded every day. Generously assuming something like 1 GB/hour for 1080p, that's 200TB per day that you have to add. No home array can handle that.

Don't worry, only a very small subset is worth keeping

That’s what you think now.

Even now, the most garbage youtube video out there is still probably useful as training data for some AI (maybe even to generate horrible youtube videos)

So true. I look at photos I've taken in the past, and discovered that I took pictures of all the wrong things.

This is a weird hackernews phenomenon where two sides of a discussion present the technical aspect of the thing they want to do, and are correct in their description of the technical aspects, without addressing the fact that they are talking about accomplishing totally unrelated objectives.

It is probably possible to horde more Youtube videos than you could ever watch, probably including most of the ones that you might ever be interested in. And it is almost certainly impossible for any individual to capture every video which goes through Youtube.

Neither of these seem to address the issue of whether there exist videos which will retrospectively have archival value which are not captured.

> Neither of these seem to address the issue of whether there exist videos which will retrospectively have archival value which are not captured.

Unless the entirety of youtube can be archived it's safe to assume that there will be something of value which isn't being preserved. It's an unsolved problem and not one Google wants to see solved.

> probably including most of the ones that you might ever be interested in.

That’s a really interesting question: how to determine videos that I might ever be interested in.

> whether there exist videos which will retrospectively have archival value which are not captured.

And that’s not really a question: there definitely exist videos that have a certain historical value which were deleted from YouTube, and most of them before I archived them cause I am lazy.

I would gladly pay for a personal archive.org - a solution that automatically archives each page I visited and video I watched. I guess the required storage amount will be pretty affordable.

> > whether there exist videos which will retrospectively have archival value which are not captured.

> And that’s not really a question: there definitely exist videos that have a certain historical value which were deleted from YouTube, and most of them before I archived them cause I am lazy.

Sure, but you aren't the only one backing up YouTube videos. It seems at least plausible that the aggregate storage capacity of the entire data horder community and their propensity for backing up whatever they come across could result in a situation where if something is interesting, somebody ends up capturing it, right?

There are some communities with a collaborative video index of who has what backed up.

I have 4PB, but my understanding is that I would fill that mirroring a single day of YouTube at reduced quality. The Internet Archive could surely handle mirroring older YouTube content with a large grant. But the upload rate plus video quality in recent years is definitely cost prohibitive to replicate.

The point I was trying to make is, data hoarders collectively archive a lot of data. They have interests and tend to mirror their interests from the sites. As a result, they can mirror a substantial amount of videos or knowledge from any gigantic site.

I'm aware that even a horde of data hoarders can't archive a drop in the YouTube ocean, but by archiving high-impact channels, they can backup both important and vast amounts of information.

The size of YouTube is likely measured in exabytes. I think it would be hard for any entity that was not organized and well-funded to mirror all of it, let alone make it available in a reasonable fashion.

Reply on news.ycombinator.com