Hacker News Reader

Comment by komali2 | Hacker News Reader

One the one hand it's sad that it'll be impossible, practically and perhaps even physically, to archive all of Youtube should Google ever decide to axe it, on the other hand even though I'm fanatically a datahoarder, the thought of archiving youtube just seems... exhausting? Pointless? Haha I'm not sure.

Then again just imagine how fantastically useful 40 years worth of video a day could be to a historian 200 years from now. I think the most important technology going forward will be analysis and indexing software, to make searching videos like this possible. Ted Chiang wrote a short story about that, where you could search like "that time I got in a fight with my wife about the toilet seat being up," and you'd get a video from the incident.

View on HN

It's crazy to think that 40 years worth of video can be uploaded daily, and yet that only captures a minuscule amount of everything that goes on in the world in a given day.

Yeah, that’s only a ratio of 14600 days per day. That’s 21 million 1 minute videos a day, which is a tiny slice of the population of the planet giving a brief glimpse.

On any given day approximately 23.37 million years of human experience occurs.

But there's a lot of crossover surely - families that live together we could consider "one" day's worth experience, minus the time they spend apart pooping or whatever.

Yeah that’s an interesting way of slicing it.

When people say “everyone’s different” I’m quite fond of saying things like “well, there’s 7.8 billion of us, surely we’re not all that different?”

There’s ~21,000,000 years of human life lived a day. At 40 years per day uploaded, that’s 0.000002 of the human experience for that day.

It’s interesting to think that YouTube content is only a “two in a million” reflection of human life. Averaged out, that’s about 1 minute per person per year.

Apply a similar comparison to the past and it's humbling to realize just how little we know about history, and how quickly we collectively forget the farther back you go.

Given we are living in 2020, there have to have been efforts or thought-experiments dedicated to archiving YouTube (would love to read about them if they exist). It seems 'almost' impossible like you said, even if we disregard, say channels with less than 100 subscribers (just as an example).

I was at a conference at IBM around 1993 at which a very smart person pointed out that we had reached peak “knowledge”, in the sense that the new explosion of data on the Internet, including the the new “www” meant indexing would soon be impossible. “You’d need to keep a copy of every host”. We all nodded because this seemed pretty obviously true.

It also wasn’t an urgent problem as experiments like http were unlikely to reach broad usage due to critical failures like the one way links.

I mention this not to mock your “‘almost’ impossible” statement but to point out that you’re in good company

From an indexing/search perspective it's already impossible to find some of the more obscure videos I saw in 2013 because of the relentless crap flood of "things sort of like what you searched for that are newer."

Well that and the growing penchant for revising history by destroying dissent, blacking out unpleasant thoughts and facts, and the onslaught of ridiculous copyright strikes against fair use content.

I constantly have trouble trying to find "that one tweet" from yesterday, a much less intimidating problem.

Twitter has notoriously flaky search in my experience; it is clearly geared towards discovering new content and not towards finding content you engaged with previously. This misses the point often, as more and more people use Twitter as a sort of social note taking tool.

I think this is one area where general purpose search engines are failing users currently. We should be able to tell Google "that one tweet from yesterday" and have it return something meaningful to us. Of course I understand that this is hard and becoming harder as social media companies hold on to their walled gardens, but clearly (to me anyway) the status quo is to the detriment of the user.

Doesn’t that suggest a market need for another search engine for this purpose?

Meta-search? Searching the search engine..I like it.

Metasearch engines are a thing: https://en.wikipedia.org/wiki/Metasearch_engine

Here's a nice one with many public instances you can try: https://github.com/asciimoo/searx

use the "before:YYYY-MM-dd" search parameter

... which you can combine with after:YYYY-MM-dd

It's not impossible. It's merely a matter of resources and will. Google not only has to store all that video they also need to transmit it over and over again. An archivist would only have to write to store the video. If humanity makes it to the year 3000 we're going to be pretty disappointed that all that video was lost due to a lack of foresight.

I bet the number of times the typical youtube upload is watched is zero. I think YouTube itself already fills the archival, non-transmitting role here.

Average views per video is around 10k. I don't know what the mode (most frequent number of views) is, but I'm almost certain it's not zero, since the uploader checking if the video uploaded correctly already counts as a view.

Anyway, for archival purposes you don't need to save the plethora of formats Youtube keeps at hand for each video, you just need one size in one codec, and it probably doesn't have to be 4K or 8K, which probably reduces the storage requirements by one order of magnitude or so.

I'm the highly downvoted parent that said this is possible.

It's only 45 years of video per day. That is very, very possible to save. That's roughly 600 terabytes of storage space a day. Which means that per month you're generating a long term storage bill that is about $18k more per month.

So after a decade of operation your monthly bill is about $2.1m which is far, far, far cheaper than what we spend on plenty of things. Like I said above: This is not impossible. In fact, I bet the NSA is probably already doing it. At least for public videos. This truly is peanuts.

Is the average as relevant as some other statistical measure that I'm not knowledgeable enough to know about? For example if the vast majority of views are on 10% of the videos, the other 90% could be stored on less expensive hardware that has less bandwidth maybe?

edit: for example the average net worth of an American is 76,000$ but toss a couple billionaires in the mix, whose net worth is larger by such an enormous factor, that the "average" is misleading

You’re thinking of the median (the value that ends up in the middle after sorting), which is indeed more useful in such cases.

(Of course, boiling a wide distribution of values down to a single value, no matter by which process, is fraught with problems.)

Well, to show the video to the person who uploaded it you don't need to reencode it or push it to the edge or any of that. And I see no reason to believe that most videos are even watched by their creators.

We get time travel by 2034 so no issues - https://en.wikipedia.org/wiki/John_Titor

There is nothing impossible about it. We barely scratched the surface of data storage mechanisms. All it takes is another few breakthroughs and we likely can store all of Youtube of all times in a block the size of an ice cube.

What makes you so confident? SSD tech is experiencing scaling challenges¹. IIRC DNA’s supposed to be incredibly dense and last long, but too expensive for much more than wedding photos². Do you have a more promising emerging tech in mind?

1: https://news.ycombinator.com/item?id=23393293

2: I think I heard this use case predicted on some NPR thing I can't find, but this one https://www.npr.org/2013/01/24/170082404/shall-i-encode-thee... says “Agilent waived the cost of DNA synthesis for this project, but the researchers estimate it would normally cost about $12,400 per megabyte.

"It's an unthinkably large amount of money ... at the moment," Goldman says.”

>$12,400 per megabyte

I believe the first hard drive cost about $100K/MB in today's money.

The Sandisk microSD card SDSQXCZ-1T00 rated at 1TB for MSRP of $400 is listed at sdcard.org as physical dimensions of 11 x 15 x 1.0 mm, Approx 0.5g. That is 165 mm^2. If a medium ice cube is 25 x 25 x 32 mm for a volume of 20,000 mm^2 - then today an 'ice cube' could hold 20,000 / 165 = 121 TB with normal off the shelf accessible technology for $48K USD.

Given that ~1 hour of video is uploaded to youtube per second and assuming 5mbit/s for the video data, then you can store under a days worth of youtube videos on that current-day ice cube (it will be full in about ~15 hours worth of uploads (if I didn't miscalculate)).

Don't forget that Google stores probably at least 3 copies of each for redundancy and multiple sizes.

Wouldn't want your only ice cube to melt

But also under aggressive and exotic compression.

Reply on news.ycombinator.com