Hacker News Reader

Hacker News

Comment by cobookman | Hacker News Reader

Parent Full thread

cobookman

You overestimate the cost of storage at this scale.

It's totally worth them storing the data given the marginal cost.

A 10TiB HDD is only 150$ retail. These videos can likely be stored in less than 200MiB on average.

Storage has gotten cheaper YoY

View on HN

Just a few days ago there was a discussion about how Google is now indexing less contented from a years ago[1].

If Google can't keep indexes which are relatively inexpensive why should they keep videos that are more expensive and possibly no one will watch? Especially as the length and number of videos is increasing faster than ever?

[1] https://news.ycombinator.com/item?id=19762907

You're not scanning through bulk storage like that all the time. Indexes have to be managed to be useful too which is another cost.

Okay, so the videos may become preserved but inaccessible.

Videos still need to be indexed, why would Google keep the videos without indexing them?

Let’s not confuse the videos with the index of the videos. Videos can be stored on the slowest and cheapest medium, indexes might be kept in memory or flash. Each video takes hundreds of mb compressed, an entry in the index could be constant per video

Indexing and storage are two different things. If something if not accessed regularly, then you don't want to index it. You can still access it, it just takes longer to find.

The Google indices are very expensive to create and serve. For one, they use a ton of memory.

YouTube has TPMs whose only focus is making storage very cheap.

Google might have thrown out some old pages, but the internet keeps growing exponentially. I highly doubt that Google is indexing less pages than a couple years ago.

I'd like someone to do the math on that 200MiB estimate. YouTube stores every video at multiple resolutions, and has replicas in data centers all around the world.

For low-traffic videos they likely only keep a copy in a single region.

That, and I’m sure they employ “cold storage” using slower (and cheaper) spinning disks.

Interestingly they could also just keep the best resolution version of rare videos, and have really powerful encoders if anyone suddenly asked for a lowres version. Then they can deliver as well as cache the lowres version in case someone else wants to look at it an hour later, but maybe drop it after a week of no access.

They don't even have to be all that powerful. GPU transcoding is fast as heck. You really just need to decide where the line is between the storage tradeoff and needing a bigger GPU farm to handle more frequent transcoding requests. I'm betting you could get away with having just one copy for 90-95% of Youtube's content. There is an ocean of video on Youtube that is accessed less than once a month or even once a year.

Ah! Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data.

But, I'm not in industry, and the last time I poked my head in I recall things being way uglier and more complicated than I had imagined. There was a good conference talk that was linked here once but I've lost it. Talked about the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything.

Interesting, thanks! I'd love to read more about ways to encode video that allows streaming at a variety of resolutions with a minimum of redundancy, if anyone has any links.

For me, it came up in a course I took last term which used the textbook "Video Processing and Communications" by Y. Wang, J.Ostermann and Y-Q. Zhang.

It's a bit out of date (2002), but the core video encoder material was solid. It's a bit heavy in representing concepts using symbolic/equation notation. That, for me, made the jump from 1D signal processing to multidimensional signal processing tougher than it needed to be.

There are probably better resources so definitely don't just take my word for it! :)

(For scabilility, IIRC the terms "enhancement layer" and "base layer" are particularly important, as are the block diagrams that generate them.)

> Small note. There are methods of compression and encoding that allow for scalability. There's some fancy signal processing you can do to encode multiple <framerates / resolutions / compression qualities> into a single bitstream without necessarily storing redundant data.

A big note. All of this wasn't used by YT because they've always used a widely used standard codecs and media containers. All the lower qualities of videos they provide use storage nearly the same as the highest one (so 2x total), all bitrates are the same for all videos. As for download part, the players in auto quality would get first a maximum quality they can for current download speed, if there's a room or connection isn't stable - also 480p version, and if the speed was enough to download current and next chunk in lower quality it would download the best and switch to it over on completion (there were a times when player was downloading the whole video in the best quality after it had finished in an optimal).

> the sort of awful, buggy things (formatting/file wise) that people try to upload that break everything.

YT has very narrow list of formats allowed to upload with hard breaks in converting process, especially for the audio.

You are correct about the cost of storage. However that is not the cost of storing these videos. When you set up a server you just buy a hard disk and some other hardware correct. But Youtube also has to buy staff to maintain, update and services and networks, offices for them, legal stuff, free food, insurance, electricity and every other thing a huge business needs of course.

Google after 2007 has been about doing things that make business sense. Just storing these videos is a cost. It's why there's been a glacial pace in new features, APIs and support in youtube for over a decade - it's not a financial priority.

That was retail price. Most the operational aspects at google are somewhat fixed costs.

Power Cooling are per Server. However given the low load on longtail video content & ability for google to co-mingle high IOPs workloads with low IOPs workload.

Its not that simple...

TLDR, storage is fairly cheap in scheme of things. Even if having infinite video storage costs 100mil/year, google likely makes more from content monitization.

These videos are hours long, and unless your doing low quality 480p, which will make it hard to follow, would definitely not take 200MB.

There is an incredibly long tail of videos that are (almost) never watched. All those 200mb videos add up to a lot of money.

> A 10TiB HDD is only 150$ retail.

Where is that? I can't find anything so cheap.

https://www.bestbuy.com/site/wd-easystore-10tb-external-usb-...

The bizarre thing is I can't find bare drives cheaper than that on Amazon - how are they making a profit on these?

Maybe it's one of those drives that comes with a demo of the backup software and they're hoping you'll buy the full version?

Generally speaking the drives in external enclosures are from the bottom slice of the still-viable drive pool. You'll often find that they have shorter/less comprehensive warranties.

Larger market of average, undemanding home users for external hard drives.

Smaller market of more demanding enthusiast users for internal drives. (Larger businesses aren't sourcing drives from Amazon or Newegg, unless the company is called Backblaze.)

Latter costs more to support and as niche users generally pay more. (The more price conscious resort to shucking external drives.)

At least, that's how it's been explained to me. I don't know if that's the entire story though it sounds reasonable enough.

How many megabytes of storage are subsidized by, say, 100 views per year?

Maybe I'm mistaken, but my understanding is that YouTube has been operating at a loss for its entire existence.

Sources:

https://www.wsj.com/articles/viewers-dont-add-up-to-profit-f...

http://fortune.com/2016/10/18/youtube-profits-ceo-susan-wojc...

https://www.bloomberg.com/news/articles/2019-02-04/alphabet-...

While there's nothing concrete concerning how much YouTube costs and brings in, it seems to be generally understood that YouTube is an investment that hasn't reached profitability yet and only exists in its current state due to the fact that Google is willing to lose money on it in the mean time.

Reply on news.ycombinator.com