Hacker News Reader

There is the rather intriguing prospect of NVM Express over Fabrics (NVMe-oF): https://en.wikipedia.org/wiki/NVM_Express#NVMe-oF

Marvel Octeon 10 DPU (with an integrated 1 Terabit switch): https://www.marvell.com/content/dam/marvell/en/company/media...

Probably pretty soon you'll be able to chuck in a few hot swappable 100 TB Nimbus exadrives (https://nimbusdata.com/products/exadrive/) in there and call it a day. 1T in 1U. :)

Interesting to see that Infiniband is still kicking

Not really. Ethernet and Infiniband are both perfectly capable from a bandwidth perspective. Streaming video isn't remotely close to latency-bound, which is where Infiniband would be better suited.

Streaming video is about the perfect thing to send as you can cache it for days.

If only it could start playing faster.

> You could ship a Rack to every Region, State, Nation, Jurisdiction or Local ISP and Exchange and be done with it.

And when a single rack is down the whole region might be impacted negatively. The blast radius of such a setup would be huge!

As a CDN you really want to avoid this. Even for regular operations you need to be able to afford some servers going out of service for software update and other maintenance - without impacting availability, latency for users (the latter would happen if you route them to a complete different region), or infrastructure costs (which increase if you can't serve data from as close to the user as possible anymore, and have to pay for additional networking fees).

All major CDN providers have hundreds of regions (all with multiple hosts), and you can't really avoid the former. You could run less hosts per region, but whether 1 is the feasible will depend a lot on other parts of your system.

> That is going from 16Tbps to 32Tbps per Rack ... only need 200 racks

I doubt ISP's give an entire rack to Netflix. I wouldn't be surprised if they only get like 4U total (hence why throughput per server is so important to Netflix).

Why not? It's top bandwidth consumer for a retail ISP and surely any reasonable amount of rack space is worth the savings in interconnect bandwidth.

They are frequently rack space constrained, hence these super dense hardware.

Some ISPs give a full rack, some don't. It depends on how much traffic they have and how willing they are.

But a lot of the racks sit at internet exchange points, where Netflix rents one or more racks at a time.

The minimum requirements

https://openconnect.zendesk.com/hc/en-us/articles/3600345383...

I think it depends on the size of isp, probably a rack would be too much even for the biggest isps, but one 4u too less.

Looking at the banner pic on their main page, they seem to have at least one ISP install of multiple racks in the wild. Also, doing a little reading on how "fill" of the devices works, they talk about doing peer-to-peer filling of appliances located at the same site, which leads me to believe, even if not deploying a full rack, deploying multiple appliances to an ISP site is a relatively normal occurance

https://openconnect.netflix.com/en/peering/

You are probably overestimating Netflix traffic by a lot.

IX.br peak traffic is 20Tb/s, DE-CIX peak traffic is 14Tb/s, AMS-IX is around 11Tb/s.

The 800Gpbs machine is probably enough for a country.

Netflix traffic stats at PIT Chile, this is their only peering connection in Chile: https://www.pitchile.cl/wp/graficos-con-indicadores/streamin...

This assumption misses out on all the private interconnect links and deployed OpenConnect appliances within ISP networks - a majority of Netflix's traffic today. IXes are only a small portion of overall internet traffic.

I notice people streaming in very low resolution without realising it, and sometimes intervene when the pain gets too great.

I’d be vey surprised if the average bitrate was anywhere near the appropriation.

However that wasn’t the point of the calculation, it was looking for a maximum.

Agree and 20 mbps is a reasonable rate for modern codecs for resolutions up to 4k for the 99% of viewers

Netflix 4K is pretty low bitrate. I haven’t checked recently but was not seeing anything approaching 20mbps last time I looked. That said, it looked pretty good.

You still need the premium package for this. It maxes at 1080p with no HDR on the standard package.

Back of the Napkin Zen4 / Genoa gets you to ~500GB/s PCIe and ~500GB/s of DRAM bandwidth -- nearly 4Tbps! Zen3/Rome is ~300GB/s PCIe and ~300GB/s DRAM -- about 2.4Tbps. A single 2U box with Genoa might scale to 1.25Tbps+ of useful Netflix traffic. We'll have to see what magic Drew can pull :)

That is less than 200 Racks to serve every single of their customer on planet earth. ( Ignoring Storage. )

If you're going to ignore storage, Netflix could just ship a low-end video server to every one of its customers and be done with it.

Every problem is an easy problem if your pretend the hard parts don't exist.

How much storage does Netflix actually need for its whole library?

It's got about 17,000 titles globally [1]. If they have copies in SD, 720p, HD and 4k that would be 68,000 versions (plus some extra audio tracks for stuff dubbed in multiple languages, but I suspect this is fairly minimal in terms of storage though)

Let's assume that the resolutions have the bitrates at 5, 10, 15 and 20 mbps.

The average length of a Netflix original movie is ~90mins [2]

So that would require about 575TB in storage if I have done my maths correctly.

You would need about 20x30TB Kioxia CD6 SSDs for all that. Very expensive but definitely technically possible.

I could totally see it being possible to fit those drives in a single node to push the 800gbps required, not increasing the over rack requirement at all. Not sure if the bandwidth from that many drives is enough, might have to cache some of the most watched stuff to ram)

Not gonna see any in home boxes with all the titles pre loaded any time soon though. As a hard drive array that's still 30x20TB drives.

[1] https://www.comparitech.com/blog/vpn-privacy/netflix-statist...

[2] https://stephenfollows.com/netflix-original-movies-shows/#:~...)

> 575TB

I wouldn't be surprised if there were a few bored and cashed up engineers with private plex servers this big out there. 575TB is way smaller than I would have guessed, although 575tb of SSD is still much more expensive than 575TB of HDD.

I guess it makes sense - Netflix' library isn't that big.

In the article you link it refers to Squid Game as "just one of the platform’s many titles" so that minutes-per-title estimate is probably super low since series are gonna be heavy hitters there compared to films.

It's probably larger than we'd guess (there's likely a lot of device-specific stupid codec profile crap that may cause extra copies too).

And - even more interestingly to me - it's not a stationary target. Gotta take mobile consumption into account too!

Do they keep on every server the global library? I guess they partition it geographically.

In their OpenConnect network they keep the most demanded titles and the latest releases. And IIRC that refreshes nightly (with new releases and whatever is hot that day)

https://openconnect.zendesk.com/hc/en-us/articles/3600356180...

Netflix is more likely to use a single box of this kind of throughout at any given POP than a rack of them. For bigger installations they can use cheaper, less throughput-dense hardware (although I don’t know if they do).

Take a look at the hardware, it isn't particularly expensive stuff.

Aside from GPUs, I'm not sure how you would increase the cost density much. Those NICs doing hundreds of Gbps and TLS aren't cheap, nor are the fast SSDs needed to sustain the load, nor is RAM or top end AMD server CPUs. Of course, the cost is absolutely worth it to Netflix!

Yes, but it's still just one box, if you're building a cluster of cheaper characteristics you need more of everything. A high-end server VS a cluster of 10 machines, 10 machines wouldn't be cheaper to get to the same throughput, it's not alien specialized supertech, it's just top of the line commodity hardware. (10 is just an example number here).

I mean, I guess I disagree with your stipulation that you couldn't lower total costs somewhat using slightly more slightly lower end hardware, if rack space was cheap.

> top of the line commodity hardware

Yeah -- cost in commodity hardware scales super-linearly with performance.

On one box you don't have to buy a 15000$ (or two) network switch(es), that's significant, you don't have to pay for N chassis, motherboards, nics, NVMe.

Considering they're really eeking every last bit of juice out of the box, I'm doubtful that distributing it would be cheaper.

Also they're not maxxing the CPU out, they just need memory bandwidth and the Mellanox nics. Storage would be more expensive on more boxes since they can't distribute the storage, they have to use local storage to reach the performance we're after.

> Storage would be more expensive on more boxes since they can't distribute the storage, they have to use local storage to reach the performance we're after.

Not necessarily. To a first approximation, if they can get 800Gbps out of the disks in a single box, they could split those disks over say 8 100 Gbps boxes and get the same performance out of the disks for the same price. Once you split it into 8 boxes, maybe instead of each box getting 2x 16 TB pci-e 4.0x4 drives, you get them 4x 8 TB pci-e 3.0 x 4 drives. Half size and older generation drives are likely to be less than half the cost. Netflix does have ways to segment cache among their appliances, so they wouldn't need to have the same storage capacity on each box as they do on the combined box.

It's certainly a procurement analysis to see if those savings will add up to overall savings, and there's a good chance it won't; especially if you need to add a 800 Gbps network switch. You do often get a pretty good cost savings by having two single socket servers vs one dual socket server though; again though, probably not if you have to add a switch.

The people doing this might also be doing infra as code for the virtualization layer on the hardware itself - which this might not be able to satisfy. At minimum they surely have a ton of this stuff deployed already so changing hardware specs big time might not be worth the cost.

Also are you taking into account encryption for those specs?