Hacker News Reader

Comment by kmeisthax | Hacker News Reader

For the streaming factor specifically, I can at least offer something resembling an answer: Google. In the early 2000s they bought up a bunch of dark fiber and peered with all the major US ISPs, and they were able to do this because no ISP wants to be the one that blocks or degrades Google. As a result they were able to host video streaming on their network without immediately being shut down by Comcast and co. Instead they had to go after Netflix.

Google has a lot of custom encoding silicon, too, AFAIK.

Storage is the biggest question of the three. Linus Sebastian specifically called this out when YouTube started really pushing to make the non-Premium experience dreadful. There isn't really some secret special sauce you can buy or make for storage. Literally everything is being stored with the same hard drives, SSDs, discs, or tapes you can just go out and buy. The only specialization you can do is build or buy equipment to handle extreme numbers of them. Google does buy these in bulk, so they probably get a discount on storage, but it's not something that would make storage costs just go away.

View on HN

> There isn't really some secret special sauce you can buy or make for storage.

Well, there's this:

https://www.microsoft.com/en-us/research/project/project-sil...

The bandwidth costs are the key. Good luck getting rates anywhere near what Google’s effectively are. Spoiler: you can’t. You probably can’t realistically get to 5x their costs, byte-for-byte.

Which makes competing with them effectively impossible except for a very-few other megacorps.

Bandwidth costs are actually free, so this isn't exactly accurate.

Most bandwidth is via settlement-free peering with thousands of ISPs around the world. At least that's how we did it at Twitch, and how we did it when I worked at a large CDN before that. There are still costs for backhaul, interconnect, colocation space, dark fiber, network hardware, and transit to fill the gaps. But this talk about how "Google can magically do it 5x cheaper" is nonsense.

Don't Comcast and friends throttle any peering points you use, until you hand over $x per subscriber per month for them to stop doing so?

Typically the problem is that Comcast won't peer with you. They always use the excuse that they only peer with equals, and since they aren't sending you as much traffic as you're sending them, it's not in line with their peering policy. This, of course, is a problem of their own creation; they're cable so customers all have a tiny amount of upload bandwidth and a large amount of download bandwidth. It is unlikely their policy permits peering with anyone. The ultimate effect is that you have to buy transit from a Tier 1 ISP instead of with the consumer ISP directly, costing you money. There are, of course, backroom deals where they sell a subscription tier that doesn't include video and then they throttle all the video. That's different than the peering issues; there is enough capacity to send all of your packets to them, they just throttle them on their end to squeeze money out of their customers.

I've worked at 2 ISPs and we obviously didn't have this peering policy, because it's dumb and it breaks the Internet. We also didn't throttle video, because it's dumb and breaks the Internet.

> There are still costs for backhaul, interconnect, colocation space, dark fiber, network hardware, and transit to fill the gaps.

Genuine question - aren't those gaps essentially what make a video streaming service operate at scale though? It'd be like saying "ya this bus can get everyone from NYC to Philly at $10 but doesn't stop anywhere in between", or am I missing something about all of those gap filling components?

To be fair, if they said "ya" they're probably in Scandanavia...

That isn't free. For every terabit of bandwidth, you have to physically build out a terabit of network. Not even remotely free. Having already built the network, and being already paying to run it, you can then use it for free, yes.

This is the model for all networks, indeed, most businesses - they pay a big upfront and moderate recurring cost to make a fast network (or restaurant or widget factory) and then sell it in slices with a large freedom to choose a pricing model. Pay per terabyte is a pretty reasonable way to pass on the network's fixed cost to consumers, just like part of the cost of the restaurant meal covers the interior decorations, even though the decorations don't actually cost more the more people eat, until the restaurant gets so busy it needs to expand.

  > For every terabit of bandwidth, you have to physically build out a terabit of network.

A lot of the content will be cached at/near the edge. I imagine a lot of time is spent watching popular videos.

Cached with what? The $0 cost hardware you gave to the ISP for free? You expect every ISP is just going to give you free CDN services?

I didn't say it's free of cost. My point is that a terabit of bandwidth delivered to consumers in a particular geographical area doesn't require a terabit of bandwidth from the center to that area, because much of the content can be cached.

So the 'terabit of network' the content provider needs to build need only span a few hundred feet within a single building.

I never said it did. You still have to build whatever you want. A terabit of edge network is a terabit of edge network that has to be physically built.

Not all networks are the same. Some have terabit backbones and gigabit edges, some have the reverse. We'd still call both of them, roughly, terabits of network, and you still have to build them. The one with the terabit core might actually be easier because you have less of the expensive really fast equipment.

OK I don't think we're disagreeing on any factual point.

The only point I wanted to make is that the 'terabit of network' doesn't have to be end to end, so it's not as scary as it may sound.

In some ways it's less scary: no long inter-site fiber runs that I assume are an absolute nightmare, and no renting those same runs at exorbitant rates. All your hands-on work remains in the datacenter, which is set up to make it a breeze. In other ways it's more scary: you have to negotiate with a lot more counterparties and visit a lot more datacenters.

I think vundercind was implicitly including Twitch in the phrase "except for a very-few other megacorps".

If you're wondering, if you can't get peering, you wind up like Twitch. In South Korea. South Korean telecom law explicitly shifts the capital expenditure costs of ISPs over to other online services, which is sort of like the fucked-up opposite of Net Neutrality regulation. So Twitch was being bled dry to pay for the chaebols' network expansion. Hell, even after Twitch left, South Korea fined them for leaving!

> Hell, even after Twitch left, South Korea fined them for leaving!

...is there some reason for Twitch to pay such a fine? Were any grounds stated for it?

> In the wake of them ceasing service, Twitch has been fined for 435 million Korean won – but not for the entire service being terminated. This is related only to them making it so users in South Korea can’t access VODs on the platform, something seen as a direct violation of South Korea’s telecom laws by Korea’s Telecommunications Commission (KCC).

> According to Yonhap, the KCC made the decision that Twitch terminating the ability for users in South Korea to access VODs wasn’t necessary to keep the service alive. When asked to justify their claims, Twitch declined due to contractual obligations related to keeping user and site data private.

> Additionally, Twitch would have to present evidence that their decision to gradually take features away from South Korean users & leave the country was necessary. This means that Twitch isn’t likely to return service to South Korea any time soon.

> There’s also a good chance Twitch will be forced to provide refunds for those who have been affected by the service being discontinued, with the KCC warning Twitch that they need to prepare “user protection measures” as they cease service in the country.

https://www.dexerto.com/twitch/south-korea-fines-twitch-over...

Is there even an enforcement mechanism other than arresting executives who go to South Korea or seizing assets that are there?

laughs in Australian My local DC charges $333 for 10TB (upload/download inclusive).

Bandwidth costs are really not that bad.

95th billing, adaptive, progressive playing and just cap buffer to the minimum to keep playing. Equals ~$1M/month for +10 tbit/s egress.

Source: Worked at one of the largest bandwidth consumers in the world.

If bandwidth costs were very high a whole lot of the services we love would cost more or not exist.

I am so confused here. Either I’ve been doing high-bandwidth bit slinging extremely wrong for quite a while or a lot of HN has never done it at all and is opining on it anyway. It’s real money, IME.

Anyone who says "bandwidth costs are not really that bad" should spend 2 minutes playing with the AWS cost calculator.

You would think the VMs are the expensive part, but no, egress is easily multiple times the cost of the compute.

To be fair, that is the penalty of being in a shared cloud. They are incetivized to keep their customers from using everything, everywhere, all at once.

Jump into a 'bare metal' datacenter and things can get much different.

Not even AWS. Any CDN. It won’t be AWS-bad (way, way, way under) but it ain’t good.

Cloud bandwidth pricing has nothing with do with costs and everything to do with lock in .

You can get 100x cheaper and unmetered at a low cost provider like OVH or hetzner or similar bare metal data centers .

It doesn’t even need significant monthly commits to get that pricing if you are running video streaming at scale you are not running on AWS or even tier 2 like OVH for sure

Are they actually unmetered or are they unmetered*?

* up to 2TB per month

It's unmetered*

* if our upstream is saturated we're going to look at our biggest users and if the number is really big we'll send them a polite email to please reduce it or pay more.

There are reports of people getting emails from Hetzner after sending multiple Gbps continuously for several months. That's the level you have to reach before the * kicks in. Only 1Gbps servers are unmetered, so you'd have to have several.

If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that.

I have one more interesting data point to add: I was quoted 950€/month for a dedicated 10Gbps between Berlin and Amsterdam (about 600km) plus peering at AMS-IX, or 300€ for 1Gbps. (They're not secretive and you can just ask for a quote using their sales contact form). Extrapolating, it seems that 1€ is worth about 2.5 petabyte-kilometers, at least within the dense interconnections of continental Europe. About twice the price of shipping a petabyte of hard drives the same distance.

> If you want to know a better approximation of their true cost just look at their non-unlimited plans: 20TB/month included for free; 1€/TB (excl VAT) after that.

I will point out that this is still about 50x-80x cheaper than Amazon. Not far off the claim of 100x.

Mhm. Save literally 98% of your egress bill by avoiding AWS, GCP and Azure.

AWS is particularly bad at this for S3.

Cloudflare and most other Object Storage Providers either fully free egress for all users or at-least for inter-cloud transit so you can then put a free/cheap CDN like Cloudflare in front and not pay all that much for b/w.

AWS refuses to participate. Costs of retrieval of all data plus associated bandwidth is a so high for many people that they stick to S3 including me.

There are enterprise plans that offer significant savings if your scale is high enough.

Yes, it’s unmetered. Just colocation. You pay for the maximum rate basically not the amount sent, if that.

The cloud bandwidth you get on your VPS has nothing to do with costs at the scale where you’ve racks in data centers.

But anyone doing things at great scale, isn’t going with OVH. You could use it as an origin I goes but you’d still need CDN for decent content delivery.

Yes, at certain scale you likely have deals with many CDNs.

At even higher scale (YouTube, Facebook or even Netflix ) you are going to be putting content caching servers at the local ISP PoPs : it is mutually beneficial to do so .

So you can piggy back on the CDNs that do the same until you can afford to do so, with the same performance. This isn’t a privilege only given to those cloud providers, anyone with a checkbook and a pulse can do it.

The key point is video streaming products are not impaired because of high cost of bandwidth, that only YouTube can only afford because Google subsidizes it from other revenue. YouTube is profitable by itself, the combination of premium and ads is more than enough to pay for it.

It is hard to directly compete on long-form video because of audience and content depth advantage Google has with YouTube.

There are successful niche players who are fairly large (like Vimeo or Twitch or even OnlyFans) who focus on specific markets that don't require social network advantage like corporate or smaller segments etc.

For general purpose media, creators are going to focus on the platform with most audience and vice-versa, very hard to break that.

Agreed, I guess my point in my original comment is; yes, bandwidth is cheap but you aren’t going to compete at any scale hosting at OVH as was proposed. And you’ll need a lot more than cheap VMs too.

It was not that OVH is competitive for streaming , it is that even likes of OVH is orders of magnitude cheaper than cloud, let alone actual setups for streaming companies whose cost data no one in this thread has access to.

Anyone doing things at great scale is buying many-gigabit connections directly. Which is also 100x cheaper than AWS.

The number above was a thousand dollars per month per 10gbps, and AWS would charge more than a hundred thousand dollars at the listed $0.05/GB price.

Note: that was for a dedicated 10-gigabit link from specific location A to specific location B, plus a peering at one large IX, without any access to the rest of the internet.

Nonetheless it does give a ballpark for the cost of bandwidth being a lot lower than people think. A 10G internet connection would be cheaper to provide in some parts of that equation and more expensive in others - should end up in the same ballpark.

By "above" I meant ckdarby's number.

And honestly 10 cents per Mbps sounds kind of high for raw transit, I interpreted it as a price for actual utilization.

It was more than 5 years ago and was not a direct commitment. Another company was being used for their data centers, and this was a lease/rental agreement of equipment; Think colocation model, but where you're like 50-80% of being the main client of the data centers.

Add 10-25% profit for that company to get closer to true "raw transit" pricing from the carriers directly.

Also, can you show me any public pricing that is 100x cheaper than AWS? With same QOS! Or are you just throwing numbers around?

What specifically is the QOS for AWS?

But sure, if you want citations for $0.10 per Mbps in bulk for transit, that's easy to find/beat.

https://he.net/ "Get BGP+IPv6+IPv4 for $0.06/Mbps!"

https://www.fdcservers.net/ip-transit Europe, North America: 10Gbps $499/month

And telegeography just sells information, but they had a blog post that's now three years out of date reporting that "In Q2 2021, the lowest 10 GigE prices on offer were at the brink of $0.09 per Mbps per month. The lowest for 100 GigE were $0.06 per Mbps per month."

You need to factor in that your utilization won't be 100%, but if you're comparing 6 cents for a Mbps and and 5 cents for a gigabyte, then the exact point where AWS is 100x more expensive is when your line is 36% utilized.

You’re getting scammed with the AWS cost calculator or anything more than $20/mbps

My home connection uploads at 681mbit/s (just did the test over wifi) for 40 euros/months. At that price, I'd get 13tbit/s for 800k euros.

It's a bit surprising that you were not getting significantly better prices than individuals.

Price business pipes where they won’t cut you off for saturating that 24/7.

[edit] and that have good deliverability worldwide, no weird paths to other consumer IPs that intermittently fail to route or inexplicably have dial-up transfer speeds. And have anything like a real SLA.

I don't think they legally can in number of European jurisdictions. This isn't like Comcast overselling their copper cables - brand new fiber paid for by the EU comes with certain obligations.

Try uploading at that rate for a few months non-stop and see how long your residential ISP will continue doing business with you.

That can be taken out by a particularly unfortunate shovel or truck though, and isn’t sufficient to run a business on.

For 800k I would have 20000 different connections. This would allow me to survive many unfortunate shovels per month.

Assuming 10 Mbit/s per stream that would serve 1 million concurrent streams 24/7. If we assume people watch 2.4 hours per day on average, it could support 10 million active users. Or a cost to serve for bandwidth alone of 1 USD/user per year.

Yeah I'm coming to similar numbers. I guess this is why the Cloudflare CTO has said several times on hackernews that at scale bandwidth is free.

Pretty sure I didn't say that, but I know who did: Cloudflare's CEO (eastdakota). https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

> Source: Worked at one of the largest bandwidth consumers in the world.

Mindgeek? :-)

Just checked his LinkedIn and yes, literally lol

Oh wow, I didn’t even snoop, just a guess.

They're no longer Mindgeek - they go by Aylo now. I worked for the company that hosted PornHub.

Could anyone help me understand what this means in practice, at scale? I keep doing the math and getting numbers either way to small or way too big. If each users streams 200MB over a 10 minute session, and they arrive on the site uniformly distributed over time (false premises, I know), how much would 1,000,000 sessions a month cost? I get confused by terabits and terabytes and how "per second" bandwidth metering actually works. I keep getting to the conclusion that for $1,000,000 you could handle 16,200,000,000 sessions per month, which is more traffic than phub gets and less money than I assume they spend.

> Google. In the early 2000s they bought up a bunch of dark fiber and peered with all the major US ISPs, and they were able to do this because no ISP wants to be the one that blocks or degrades Google. As a result they were able to host video streaming on their network without immediately being shut down by Comcast and co.

This sounds like a good answer, but falls away drastically when you realise the vast majority of consumer of YouTube are outside of the USA, which in turn means so are those bandwidth costs.

Are you guessing or have I missed something here? I can't see how this could be a significant enough factor to make the global model work.

When does YouTube start deleting content?

They'll have to do that eventually, right?

So long as storage costs decrease exponentially I think they'll keep everything.

Especially if the amount of content uploaded keeps going up, so the relative benefit of deleting old stuff is small.

> So long as storage costs decrease exponentially I think they'll keep everything.

I agree. That's why I've recently made a practice of backing up things to which I'd regret permanently losing access.

https://ourworldindata.org/grapher/historical-cost-of-comput...

Relatively recently they updated their policy. If you don't login to your account for two years they will delete everything. Source: https://support.google.com/accounts/answer/12418290?hl=en

I have two accounts from my youth with mostly fansubs and "funny" vids. (A "Everybody Draw Mohammed Day"-Video that, 11 years after its upload, got band in Pakistan.) It includes two very successful (over 1M views combined) uploads of political TV shows. I will not log in to this this accounts. So this stuff will get deleted. So in the long run, stuff will disappear from YT.

I believe YouTube content was explicitly exempted from that deletion.

You seem to be right. This Google Blog post[1] says "we do not have plans to delete accounts with YouTube videos at this time."

[1] https://blog.google/technology/safety-security/updating-our-...

Would be nice if they would write that into the actual policy and don't just add a random sentence to a blog post.

If I remember right the original policy was to delete the YouTube videos and after pushback the YouTube exception was added.

You don’t need to keep everything “hot” all the time. Storage tiers exist for a reason.

My 15 year old videos with 30 views still load nearly instantly. It’s as close to hot as hot is

But they are tiny 480p files yeah?

Speed/latency doesn't tell you much, because it's all on a hard drive somewhere.

The question is whether YT is serving up the one (redundantly-backed storage) copy they have of your almost-never-watched video, or whether it's serving it up from one of 1,000+ copies it's made across the globe for currently popular videos.

Think about what you actually need to start a video. Maybe a dozen MB?

After that, you can plunge into colder storages and warm things up as you stream. Additionally, if you need longer to 'defrost' things, just cache a few more MB at the front. Cheat a bit by assuming 480p to start with if you need to; even less to store.

There is also location location location.

Maybe Google holds your content in 7 data centers round the world (~1 per continent for planned maintenance + latency + reduced oceanic fiber usage).

But with old rarely streamed content they might cut that down to just 3.

That doesn't match my experience. I have some unlisted videos that I or a small handful of friends might go back and watch once a year, and it takes several seconds of loading before they start playing. It's very noticeably different from the near-instant loading of most videos I watch.

Exactly! This was my go-to approach for reducing storage costs. Customers don't get spooked when they get an extra 1s delay for something they search once in a month. However, an extra 30ms delay in "everyday content" is a sure way to loose your users.

However, implementing this in practice is non-trivial. Knowing what is "everyday content" versus what is "once a month content".

To add more complexity -- you have these semi-predictable hype-waves especially two peaks in case of most YT videos where a "once-a-month" content becomes an "everyday" content before again becoming a "once-a-month" content. It feels you could specifically optimise for this -- reduce storage costs without sacrificing UX.

> To add more complexity -- you have these semi-predictable hype-waves especially two peaks in case of most YT videos where a "once-a-month" content becomes an "everyday" content before again becoming a "once-a-month" content.

Caching is hard but this sounds like an ARC would likely catch this, if it occurs on a small number of videos concurrently.

Old content has value for training AI models even if there are no human viewers anymore.

Yes, it's inevitable.

My guess is that the first step will be to re-encode all the non-popular videos with severe lossy compression.

They still keep the original uploaded files.

You know that because when they release a new format (eg. HDR or a different resolution), they re-encode from the original. Various people have tested that with moire patterns and various other ways to demonstrate if something was encoded more than once.

I had an entirely private channel with a couple hundred 1-3 hour long HD recordings of video conferences / screen shares. Entirely private as in every video was private, no one but me ever had access to them. Last year the channel was nuked and the account banned because I apparently broke their rule on “impersonation”. Seriously. I could file a dispute, but in typical Google fashion that dispute disappeared into the void. Thankfully the YouTube ban didn’t affect other aspects of that Google account, and didn’t spread to my other Google accounts. I was worried for a minute since I have another account with >10k subscribers, and while I don’t do monetization, it would suck to have that nuked as well.

I suspect that’s Google beginning to trim unprofitable channels using a lot of storage: delete them for bullshit reasons.

Once AI generated video gets particularly good, they could probably save a lot of storage by 'compressing' videos into textual descriptions and a few key screenshots that are reimagined on demand by the AI

getting closer to mythical storytelling of old every day

What you are describing reminds me of Nvidia DLSS3 AI frame generation. Isn't the storage cheaper than compute, especially graphics?

Is there a way to run these things deterministically with a seed? Totally AI-generated videos could just be compressed to text without any reference screenshots

isn't 90% of the videos compress-able to "cute kitty playing with something in a funny way"

Reply on news.ycombinator.com