Netflix begins the deployment of their own content delivery network | Hacker News Reader

Netflix begins the deployment of their own content delivery network

zdnet.com

76 pointsfendrak43 comments

From https://signup.netflix.com/openconnect/software:

Operating System

For the operating system, we use FreeBSD version 9.0. This was selected for its balance of stability and features, a strong development community and staff expertise. We will contribute changes we make as part of our project to the community through the FreeBSD committers on our team.

Web server

We use the nginx web server for its proven scalability and performance. Netflix audio and video is served via HTTP.

Routing intelligence proxy

We use the BIRD Internet routing daemon to enable the transfer of network topology from ISP networks to the Netflix control system that directs clients to sources of content.

They are probably using ZFS also. (my guess)

Is ZFS stable enough on FreeBSD? Seems to be an unofficial port.

ZFS is very stable on recent versions of FreeBSD, I can't speak for older versions as I don't have experience. I at least haven't had any issues with it and we use it in production.

It is an older version and not as feature complete as what you will find in Solaris.

ZFS is very stable on FreeBSD.

But you need a lot of extra RAM compared to the usual filesystems. Is that correct? If yes, is using ZFS worth the cost of the extra RAM?

ZFS doesn't use that much RAM if you don't turn on deduplication, which is of less use in a system that serves large compressed video files than one that handles tons of read/write traffic as an office file server.

You don't need the extra ram. ZFS uses it as a cache, more ram, more cache. That's about it, otherwise it doesn't use much ram.

RAM is not required; it just makes lookup faster with caching.

Why would they use ZFS on a CDN? IIRC, ZFS has some performance characteristics that would make it a poor choice for something like caching data on a network edge when it's really designed for ensuring data integrity.

Interesting box. With an AFRR of 5 - 7% the box would nominally 'lose' a hard drive or two a year. I wonder how they deal with transient failures (which would be more like 2 - 3 month). If they do the 'triple replica' thing they only get 36T effective per box, but I suppose at about 2G / movie in MPG2 thats still about 18,000 movies 'online' as it were. Now if they did something wild and crazy like a form of RAID6 that would be even cooler (and give them like 100T of actual space or 50,000 movies. Serious video, but really hard to stream to multiple people.

The old SGI/TimeWarner Video-on-demand nightmare seems so quaint now. That had a full cabinet (42U) of storage and servers and it poorly served about 50 streams of a couple of thousand 'titles' and cost > $2M each if my aging brain remembers all the specs. Hmm these guys [1] peg the cost for the "Full Service Network at $100M which sounds about right.

[1] http://www.vizworld.com/2009/04/what-led-to-the-fall-of-sgi-...

Since it's a cache I don't think there's a big problem in losing a hdd and some cache capacity. I would expect it to be more efficient to use as much storage as possible even if a hdd will fail.

I agree the cache nature makes it easier, but consider that a 3TB drive, at 2GB per 'movie', holds 1500 movies. So when you lose a drive, if the only content you have is on that drive you need to seamlessly switch from sucking it off the drive and sucking it down the back haul network. I can think of a number of ways [1] that might be done with varying degrees of 'hiccup' on the user end, but it is a big chunk of cache to lose at once. Note that 3TB is also 50 minutes of full on 10 GbE pipe bandwidth. But as you say it isn't as if you lose data, since its still there back at the head office, but you lose that copy of it.

[1] Ways that come to mind are; multiple boxes with a diversity of the content amongst them so that another box can 'pick up' the stream, keeping a stream to the home office open but not pulling data until you lose the local source, or even doing a 'buffer' to disk and stream from the buffer then switch disk targets when your disk fails.

I've often wondered if CDNs are going to go out of business in the streaming world, and no longer be relevant, in the world of (rapidly) falling transit costs.

Once transit costs drop below $1/mbit @95th percentile (circa 2014 [1]), it's costing you $1 to transfer approx 300 gigabytes/month. At that point, the management costs associated with all of those distributed CDNs becomes greater than your transit costs. This is somewhat offset by continuously dropping prices of CDNS [2] which is as low as $0.02/Gigabyte on a 500 TByte commit.

What this does tell me is that NetFlix thinks they have an opportunity to save money on CDNs and can beat that $0.02/Gigabyte.

Bandwidth Delay Product(BDP) - keeps CDNs relevant for _downloads_ (iTunes, Software, etc..) - because everybody likes to max out their 100 mbit comcast connection, but you don't need much bandwidth to stream a show. Maybe some value for CDNs in HD or live event streaming.

[1] http://drpeering.net/AskDrPeering/blog/articles/Peering_vs_T...

[2] http://blog.streamingmedia.com/the_business_of_online_vi/201...

The minimum for deployment of a Netflix node is 5 Gbps within a metro area. You must also peer with Netflix at a common interconnect so that "loading" is free for them.

By your math they are saving a minimum of $5k per month per deployed device.

Any network not meeting these minimum requirements will be served under existing Level 3/Limelight/Akamai contracts.

Your calculation doesn't account for the artificial barriers that some ISPs put up by charging the CDNs to peer with them (see Comcast and Level 3).

Isn't that orthogonal to the choice between serving from the edge or from a single mega-datacenter?

Cogent is already $1/mbps at "retail" pricing, Level3 (better quality) at $3/mbps or less in some cases.

Pretty sure CDNs will be around. The value add of a cdn isn't their cheaper commit rates. The value is in location diversity and capital investment in infrastructure and ports in those locations. In new speak CDNs are "bit delivery as a service". The customer pays a premium over raw transit to have the cdn make those capital & management investments.

Even for Netflix or another DIY delivery case a CDN will continue to have value. A couple cases immediately come to mind: Lower first byte latency on first video blocks, then a hinted handoff to the openconnect device. Resiliency/recovery of network events, in a similar manner. "long tail" content that doesn't fit in the open onnect cache width.

A key point of these devices is IIRC they're only available to peered ISPs in one those NA IXs. Netflix is essentially converting 5-8gbs of CDN/transit expense in to $50,000 (WAG) of capital. That's what's really brilliant to me.

Cool, maybe they can deploy it at comcast and deliver content via the same magical content pipe comcast uses for their xfinity service.

Most of the large CDN's already deliver straight into Comcast's network. Last I heard Netflix was already paying Level3 to do this. Xfinity doesn't have a magic pipe, all the content and deliver is internal to the Comcast network so its its own "CDN".

It was a joke, the hint was the use of the word magic.

More info on Comcast's DiffServ magic: http://ber.gd/post/23025893856/comcast-traffic-prioritizatio...

Which Comcast says isn't happening.

http://blog.comcast.com/2012/05/the-facts-about-xfinity-tv-a...

I don't really want to rehash that whole discussion, but Comcast admits they're using DiffServ tagging. People are just debating whether it's a form of "prioritization" or not.

I think Comcast would be pretty opposed to let a competing product into their not-for-sale DC space. The real benefit here is that it saves ISPs on peering charges. Comcast, AFAIK, does a lot of their own peering and has financial motivations to NOT let Netflix into their DCs (onDemand.)

We will see how this plays into the me neutrality battle...

Comcast does not operate their own video delivery infrastructure. They contract that out to Akamai. Furthermore, Comcast will happily sell non-transit, A.K.A. paid peering, to all who come. They don't (and never have) sell colocation (DC space). Presumably, Netflix is just going to buy access to the Comcast network, same as every other CDN has done.

http://www.comcast.com/dedicatedinternet/

Makes me wonder how long until Netflix looks at doing managed services for video delivery along the lines of Amazon's AWS services.

But would that have great business potential to them? For Amazon, having that infrastructure helped create not just future products (Instant Video, Amazon Music Cloud, Amazon Cloud Drive, the Kindle and Kindle Fire backend systems), but was aligned as an additional revenue source.

You'd have to wonder if enough people would want to use Netflix's technology stack in a way that didn't include distribution on Netflix.

The entire television/sports/web video/publishing industry may be able to find a use for such a thing

That's valid, except most of them have their own solution partners. What I mean is, does Netflix want to invest the resources that would necessitate making this an offering for all but the biggest customers.

Keep in mind, Netflix is one of the BIGGEST online video providers, period. Very few companies are going to need some sort of solution at their scale -- and those that do, likely have their own solutions. That means, your target market are smaller companies.

I just don't see Netflix interested in getting into a space that Brightcove and 5Min already dominate on the smaller web side -- and that custom Amazon, Level 3 and Akamai contracts take care of on the high end.

Netflix built this solution because it hopes that it will save them money and resources, somehow, I doubt that it's s scalable model (at least in terms of Netflix's required investment for support and sales) to offer to other partners.

I could be wrong, I'm just not sure I see the potential for a company that's biggest competitor is HBO.

Netflix has a very different attitude than Amazon; they wanted to spin off DVD rental because it was distracting from their real business.

Hopefully they should be able to use these cost savings to purchase more content and get it into their system. Does anyone have any informed speculation about how much they could potentially stand to save? I'm unfamiliar with how peering agreements work and or the cost structure.

It's interesting from a bit of a game theory standpoint. The optimum outcome might be achieved if Netflix hands over the appliance and the ISP installs and keeps it running at no cost to either party, except for maintenance costs (hardware replacement, etc.) which Netflix bears. Both parties save.

If Netflix wanted to get clever, they could calculate the cost an ISP would likely pay for the bandwidth their users consume, and charge the ISP say 40% of this amount to host the appliance. Netflix saves 100% of its third party CDN bandwidth costs for that ISP and the ISP saves 60% of their peering bandwidth costs.

If the ISP wanted to get clever, they could calculate the bandwidth that Netflix is paying a third party CDN for the traffic consumed, and offer to host the appliance for 40% of this amount. The ISP saves 100% of its peering bandwidth costs for Netflix traffic and Netflix saves 60% of their CDN cost.

What's likely to happen is a bit of cat and mouse over the costs. The ISP peering is likely a lot cheaper than Netflix's CDN costs, so this could work out as:

    Netflix CDN cost
  - ISP peering cost
    ----------------
  = Cost to Netflix

This is all pure speculation with made-up percentages, of course.

comcast is actually charging people more than transit rates to get access to their network -- which is why (except for level3, who has a special deal) video on comcast sucks even with large comcast pipes.

Actually, no, Comcast's paid peering costs are significantly cheaper than all but the lowest quality transit. In other words, if you just have Cogent, you get what you pay for.

That may be true at the 10G level (which I didn't price), but isn't true (in my experience) at 1G. Plus, it's a lot more likely that you'd want 10G of transit and 1G of comcast than 1G transit/10G comcast, so per-G, your decent quality (Level(3), nLayer, etc.) transit is going to be cheaper. Comcast also seems to negotiate less, since they're effectively a monopoly on routes to themselves, vs. transit.

(Obviously the right thing to do is grow to needing 10G of comcast and 50-100G of transit...)

Fair enough. The last time I was privy to this, it was more than a year ago. I didn't expect things to change that much :-).

Netflix already said the box is free (as long as you're big enough to actually need it).

I think the content question will never be solved as long as

a) Cable companies put the squeeze on content owners to give them exclusive rights

b) Netflix religiously clings on to both commercial-free and buffet-style pricing

It's tricky. You have a movie studio that pumps out AAA features that cost upwards of $200 Million to make, then Netflix comes along and says "hey, you mind taking this really low price for your content, hopefully in perpetuity?" What would your response be? So, it may be awhile before Netflix gets the big-budget blockbusters quickly. That is, unless, their subscriber base goes over 50 Million, then by ALL means game on.

I think that, in the interim, a dynamite move for Netflix would be to acquire a music streaming company like Pandora or Spotify, then a streaming game company like Galkai or OnLive. Push it out on a set top box, a la the OnLive console, charge $19.99/month with the opportunity to rent premium newly-released blockbuster movies, hot records, and AAA games on the day of release for the same prices that redbox or gamefly charges to rent them per day.

Here's the accompanying Netflix blog post:

http://blog.netflix.com/2012/06/announcing-netflix-open-conn...

I have worked on Netflix's servers first hand. After seeing how large of a presence they have in their current CDNs, I can say that they have a hell of a lot of work ahead of them for that to become viable. They are enormous.

Completely off-topic, but I love the redesigned Silverlight player for Netflix on PC. It's gorgeous.

Reply on news.ycombinator.com