Netflix's Open Connect Appliances
gizmodo.com
gizmodo.com
Comcast and Verizon now claim that Akamai is indeed paying them. Perhaps the outcry over Netflix being extorted explains why these deals have remained secret for years.
Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the random reads.
I would have assumed that having even 100 people streaming from a single box would be an interesting challenge.
> Sometimes it'll even have multiple copies of the same show, if something is in crazy high demand.
that statement from the article lends one to believe that the raid arrays are in parallel.
These boxes are meant to reduce the number of packets going all the way across the internet not necessarily remove the need.
Also, in addition to lots of storage, these boxes probably have loads of ram that they can buffer popular streams into and then serve it.
If they can cram in 256GB of DRAM, that's enough room for a 64MB buffer per stream, or about 170 seconds worth of streaming. Now you only need to be able to fill those buffers at a rate of about 24/second.
I'm assuming whatever file system is on the disks uses a massive block size, so the number of seeks you'd have to perform to pull 64MB off is probably pretty low. Eight? Sixteen? Even if it's the latter, that's only 384 seeks/second, which you could very plausibly do striped over only a half dozen disks, and the device presumably has many more.
Netflix uses UFS+J on FreeBSD 10.
Here are some notes from their talk at NYCBSDCon 2014:
- 400,000 stream files per appliance.
- 5,000 - 25,000 client streams per appliance.
- 300 - 500 streams coming off each disk all the time.
- Attempt to buffer 1MB ahead, but caching is futile.
- Result is completely random disk workload.
- System becomes limited by disk latency and CPU load.
Video here: https://www.youtube.com/watch?v=FL5U4wr86L4
I think this can be fixed by prefetching harder (like 1-2 MB).
100 Netflix streams would be less than 100 MB/s which can be satisfied with a single SATA drive. And some of the Netflix boxes have dozens of SSDs that each do 500 MB/s.
I'm not sure if the I/O is really all that random. Theoretically, it's all sequential, but might effectively hit the disk like random I/O because of concurrent streaming sessions, varying network speed and so on. My guess is you can get pretty close to sequential I/O with some simple means like using large block sizes in your RAID, well-tuned readahead and Native Command Queuing.
* Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives,
* Use 50 in a chassis of them instead of 12.
* Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid.
* Have multiple copies of a show on disk, as the article states happens.
* Optimize for a read-heavy workflow during peak hours, (eg, mount noatime).
The devil is in the details, but what I listed above is probably a good starting point. The article states a data rate of 3GB/hour, which is only .83 megabytes/second, * 100 streams is only 83 megabytes/second, which is easy for the above configuration. Hell, I'll bet the above configuration could do 10,000 streams if the data rate is 3GB/hour with no issues given Netflix's peak read-only workload.
The only possible issue of the ones you raised is low data density, but all engineering is a trade-off, 15k rpm drives get you better seek times, which would generally lead to the ability to support more viewers in a single box - not working for Netflix, I don't know how many users they want to support per-box.
It's totally possible that between the number of streams they want to support, and the total storage in the box that they've gone with SATA drives, but to pretend that 15k RPM SAS drives are "an _extremely_ poor choice" for an enterprise-grade storage system would be to ignore the fact that Netflix is making an enterprise-grade storage appliance - and that on the top-end, those appliances commonly use SAS drives.
To address your points individually:
1) Power hungry. When you add in conversion and distribution and cooling, every watt consumed by the computer is consumed again by the datacenter infrastructure. Power costs money.
2) Hot is just the corollary of 1). Hot is also the enemy of density, and this box is very dense.
3) Sensitive to vibration. If you aren't intimately familiar with this fact then you aren't getting the performance you paid for from your 15k disks. To achieve their spectacular claimed seek times then need very careful mechanical design of their enclosure. Much more careful than racking up one of Netflix's boxes in a rack with other random vibrators.
4) Density. To get the space Netflix is using here, you need 5x more expensive 15k disks because they top out at 600GB and the ones people actually use for these workloads have 3TB.
A smart read-ahead strategy obviates the need for shiny seek time specs. For any given stream you could read ahead by 32MB or whatever. Now you've made seek time irrelevant. Put lots of RAM in the machine and you're done at a tiny fraction of the capital and operating costs of 15k disks.
1) Power Hungry, the total power envelope of a rack position is the limiting issue. You can fit 16-20 disks per RU no problem. The constraint on total density is your 5kVa or 10kVa power budget. Watts matter.
3) sensitive to vibration, be very very quiet http://www.youtube.com/watch?v=tDacjrSCeq4
4) Density, actually 4TB is hitting the sweet spot for $ per byte last time I looked. If you need absolute density look for 5 & 6TB Real Soon Now. If you can tolerate some loss variable density disks around 5TB look quite a bit more cost effective.
5) Read ahead, you only need to read ahead by a couple of chunks. For video ball park it at 2MB. You don't really need lots of RAM, think 32-64GB per chassis. Additionally each 8GB dimm costs about the same power as a disk. By going with 32GB instead of 64 I can fit another 2-3 disks in the power budget per chassis.
See http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd...
http://www.listbox.com/member/archive/247/2014/07/search/bmV...
This is what game theorists call a 'pure bargaining' [1] problem:- reaching a deal makes $900, but we have to decide how to split that between us. Both of us want as big a share as possible, but our main leverage is refusing to reach a deal.
In this situation, I am the ISP; installing a Netflix box costs me $100 for power, bandwidth, whatever. Netflix are the ones that make $1000, as installing the box lets them get more customers. ISPs refuse to install boxes because they want more money than Netflix is offering, and refusing to install is the only way to get it.
For comparison, Supermicro makes one of the highest-density storage servers that I'm aware of[1]. 72 3.5" drives in 36 drive bays, so up to 288tB of raw storage, if you're brave enough to use 4tB drives.
[1] http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...
[1]: https://www.netflix.com/openconnect/hardware [2]: http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4
That's correct.
> so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed.
Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.
So not even redundancy? Just cope with losing whatever media happened to be on the dead drive?
The article also mentions the box stores multiple copies of done things for increased throughput on popular titles.
My question (without any intent to trivialize what Netflix is doing) is, isn't that just a good-old caching system?
But it makes me wonder if it would be possible to create a distributed, localized caching system? For example, if I watch The Avengers, my machine could be configured to cache a part (or all) of the movie. The more people watch it, the more it is distributed. Since Netflix can correlate location data with what movies are watched, they can target certain areas to cache certain movies more aggressively. When the next person in my neighborhood watches The Avengers, they would pull part of the movie from me (maybe the beginning until the rest buffers) and cache another part of it themselves for the next person in line. That way all the people in my neighborhood or adjacent neighborhoods don't have to pull the entire movie across the internet every time - instead, we help each other watch movies with higher quality.
On second thought, this may not be cheap considering the amount of development that may be required (and, at the end of the day, someone has to pay for this storage - although Netflix could, for example, offer personal mini-caching boxes in exchange for a monthly discount), but it's fun to dream about a sanctioned, decentralized, peer-to-peer movie service.
It _would_ be a perfect setup (much much better than sending the same stream through the same lines just because 2 persons are watching the same show)
The problem here is not technical (Bittorrent solves that) the problem is societal: for most of the content that is of interest to population, you are not allowed to redistribute it. So Netflix has to DRM-protect the content for everyone, meaning that the actual content that transits on the line has to be different, meaning that you can't redistribute it (since it's targeted to you, no one else could decode it).
If you want to see how much the technical side works, take a look at PopcornTime.
I haven't really explored PopCorn Time but I'll take a look.
With regards to DRM, the content must be de-protected before you watch it - so if Netflix installed a mini-box on my network, they could decrypt it to let me watch the show, and encrypt it to store on the device before it is sent to someone else. The key itself could be retrieved from the a centralized server.
If I think about it some more, there really is no need to keep an entire Netflix catalog in every location. As I mentioned, they have usage data and a strong prediction system too, so predicting the next show that will be watched in an area and even when would be "straight-forward" (In quotes because it wouldn't be easy for me to do it, but I think it would be easy for Netflix).
Another thing Netflix could try is pre-fetching shows during off-hours. That would work well for binge show watchers, and reduce peak time traffic.
Presumably there is limited space in the ISP data center, so they cannot let everyone who wants to stick a box in there do so, so I'm wondering what methods a common carrier can use to decide which content providers can have access, and how this fundamentally differs from the "fast lanes" that common carrier status is supposed to prevent.
From the ISP's perspective they are hosting these boxes to save money, not improve performance. If performance happens to improve (ahem) that's a convenient side effect.
There are a few other legitimate reasons as well. Some ISPs dislike hosting hardware that they don't own in their data centre. Or it may be that an ISP has contracts in place that commit the ISP to certain bandwidth costs, so removing Netflix traffic may not actually save them money.
Although this makes a fairly rich target of concentrated content ;)
" Each appliance stores a portion of the catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles being added to the service, and re- encoded movies are all part of the up to 7.5TB of nightly updates each cache must download to remain current. We recommend setting up a download window during off-peak hours when there is sufficient free capacity."
"If you had to take a 7.5TB update on your home internet, you would be screwed. According to Ookla's Net Index, the average download speed in the United States is 18.6Mbps. So with an average connection, that 7.5TB would take you about 40 days to chew through.
"Fortunately, data centers tend to have a little more speed to work with. A firehose, compared to your bendy straw. So Netflix recommends a much shorter 10-hour span (from 2AM to noon, local time) where these boxes can soak up their updates with 1.2Gbps connections. That is to say, 'Google Fiber speeds.'"
Source: TFA.
I request a stream. It's proxied through the device. It's not present on the device. So the device requests the 1080P from Netflix. It transcodes the stream it receives on-the-fly and sends me whatever quality I require while writing both the original and my lower-quality version to disk.
I mean maybe the limitation in that plan is the number of streams that could be transcoded simultaneously. So maybe instead the on-the-fly version isn't cached. Because I may need to switch quality on-the-fly. So maybe instead jobs are just queued up for the 5 or so formats required based on the 1080P download and they can process on spare transcoders opportunistically.
Just interesting to think through it as a programming problem. I'm sure the people working on this are very smart. It just seems really heavy on the bandwidth utilization as is.
But: I agree with your bigger point, that it's an interesting programming problem.
We're all familiar with CPU caches, so it's interesting mental exercise to think of this as the same system, but optimized for a different part of design space.
"It's rarely—if ever—a full copy, but rather a massive chunk specifically designed for a specific country or region, depending on what's available and what's popular."