Hacker News Reader

Comment by drewtato | Hacker News Reader

I was expecting to find out how much data YouTube has, but that number wasn't present. I've used the stats to roughly calculate that the average video is 500 seconds long. Then using a bitrate of 400 KB/s and 13 billion videos, that gives us 2.7 exabytes.

I got 400KB/s from some FHD 24-30 fps videos I downloaded, but this is very approximate. YouTube will encode sections containing less perceptible information with less bitrate, and of course, videos come in all kinds of different resolutions and frame rates, with the distribution changing over the history of the site. If we assume every video is 4K with a bitrate of 1.5MB/s, that's 10 exabytes.

This estimate is low for the amount of storage YouTube needs, since it would store popular videos in multiple datacenters, in both VP9 and AV1. It's possible YouTube compresses unpopular videos or transcodes them on-demand from some other format, which would make this estimate high, but I doubt it.

View on HN

That storage number is highly likely to be off by an order of magnitude.

400KB/s, or 3.2Mbps as we would commonly use in video encoding, is quite low for original quality upload in FHD or commonly known as 1080p. The 4K video number is just about right for average original upload.

You then have to take into account YouTube at least compress those into 2 video codec, H.264 and VP9. Each codec to have all the resolution from 320P to 1080P or higher depending on the original upload quality. With many popular additional and 4K video also encoded in AV1 as well. Some even comes in HEVC for 360 surround video. Yes you read that right. H.265 HEVC on YouTube.

And all of that doesn't even include replication or redundancy.

I would not be surprised if the total easily exceed 100EB. Which is 100 (2020 ) Dropbox in size.

For comparison, 100-200EB is roughly the order of magnitude of all HDDs shipped per quarter:

https://blocksandfiles.com/2023/07/11/disk-drive-numbers/

Sure but exabyte-level storage is done with tape, not HDDs.

I doubt any of YouTube is being served off tape. Some impressive amount of its data is probably served from RAM.

I mean, it would explain the minutes-long unskippable ads you get sometimes before a video plays. There's probably an IT maintenance guy somewhere, fetching that old video tape from cold storage and mounting it for playback.

I don’t have ads. No wait times.

> EB

I pine for the day when "hella-" extends the SI prefixes. Sadly, they added "ronna-" and "quetta-" in 2022. Seems like I'll have to wait quite some time.

> Originally 'quecca' had been suggested for 1030 but was too close to a profane meaning in Portuguese

(from https://iopscience.iop.org/article/10.1088/1681-7575/ac6afd, via wikipedia)

Seems like we missed opportunity to have an official metric fuckton.

For anyone wondering "queca" would be the normal spelling of the "profanity" although it's probably one of the milder ways to refer to "having sex". "Fuck" would be "foda" and variations. Queca is more of a funny way of saying having sex, definitely not as serious as "fuck".

Hyundai Kona on the other hand was way more serious and they changed it to another Island in the Portuguese market. Kona's (actual spelling "cona") closest translation would be "cunt", in the US sense in terms of seriousness, not the Australian more light one.

Source is I'm portuguese

> Two of the suggestions made were brontobyte (from 'brontosaurus') and hellabyte (from 'hell of a big number'). (Indeed, the Google unit converter function was already stating '1 hellabyte = 1000 yottabytes' [6].) This introduced a new driver for extending the range of SI prefixes: ensuring unofficial names did not get adopted de facto.

Rats. So close!

We should petition for "fukka-"

So is YouTube storing somewhere in the realm of 50-100EB somewhere? How many data centers is that?

> The 4K video number is just about right for average original upload.

No, it definitely is not.

For Smartphone / CE grade recorded video. Not for professional ones. Remember that number is average.

4k is not the average. The amount of sd videos out number hd and of the hd most are 1p max.

What is 1p?

1k. 1280x1024

It's not the average in either case.

I was under the impression that all Google storage including GCP (and replication) is under 1ZB.

IIUC, ~1ZB is basically the entire hard drive market for the last 5 years, and drives don't last that long...

I suspect YouTube isn't >10% of all Google.

On one hand: just two formats? There are more, e.g. H264. And there can be multiple resolutions. On the same hand: there might be or might have been contractual obligations to always deliver certain resolutions in certain formats.

On the other hand: there might be a lot of videos with ridiculously low view counts.

On the third hand: remember that YouTube had to come up with their own transcoding chips. As they say, it's complicated.

Source: a decade ago, I knew the answer to your question and helped the people in charge of the storage bring costs down. (I found out just the other day that one of them, R.L., died this February... RIP)

For resolutions over 1080, it's only VP9 (and I guess AV1 for some videos), at least from the user perspective. 1080 and lower have H264, though. And I don't think the resolutions below 1080 are enough to matter for the estimate. They should affect it by less than 2x.

The lots of videos with low view counts are accounted for by the article. It sounds like the only ones not included are private videos, which are probably not that numerous.

You’re forgetting about replication and erasure coding overhead. 10 exabytes seems very low tbh. I’d put it closer to 50-100EB at this point.

I did the math on this back in 2013, based on the annual reported number of hours uploaded per minute, and came up with 375PB of content, adding 185TB/day, with a 70% annual growth rate. This does not take into account storing multiple encodes or the originals.

Keep in mind youtube permanently keeps a copy of the original upload which may have an even larger file size

Do you know that for certain? I always suspected they would, so they could transcode to better formats in the future, but never found anything to confirm it.

I know it was true at least a couple of years ago, but there is no guarantee that they keep it.

On all of the videos I have uploaded to my YouTube channel, I have a "Download original" option. That stretches back a decade.

Granted, none of them are uncompressed 4K terabyte sized files. I haven't got my originals to do a bit-for-bit comparison. But judging by the filesizes and metadata, they are all the originals.

I'm interested in the carbon footprint.

https://sustainability.google/operating-sustainably/net-zero...

Thanks, 10 million metric tons per year. ~0.1% of global emissions.

Pretty wild stuff.

Reply on news.ycombinator.com