Netflix HTTPS performance experiments for large scale content distribution

Gmail servers probably do actual work. Netflix's CDN just copies bytes from disk to network, so TLS overhead is a greater percentage.

It's surprising. They say that they are using the hardware AES support in the CPU too. They do mention that they are no longer able to take advantage of sendfile() which was letting the bulk of the data transfer to happen completely in kernel space.

So, in their case, they had already optimized things to such a degree that they were having a lot of throughput and very little CPU usage per byte. In fact, it's possible that on their old system the data streams wouldn't have to be touched a byte at a time, as they could be loaded into main memory via DMA from the disk and once that finished they could be transferred to the network card, again via DMA. Once sendfile() is called, no further context switches are necessary for that request.

Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI. I tried running two concurrent copies and they each got that speed, which tells me that it's per-core. That's easily enough speed to saturate a 10G connection.

It's quite possible that they have much older or slower hardware involved, since when you're building a server like this normally the network or disks are the limiting factor.

If you're just pushing raw bytes down a socket, you can use sendfile(2) to direct cached data directly to the NIC without copying it to a private buffer first. This is a pretty big deal at 40Gbps, and Netflix have funded quite a bit of work towards improving it:

  https://secure.freshbsd.org/search?project=freebsd&q=netflix+sendfile

TLS kicks all that in the teeth - each encrypted stream needs its own private buffer which has to go in and out of the CPU before it gets anywhere near the NIC. That costs memory bandwidth (a mere 400Gbps with quad-channel DDR3), memory you could have otherwise used as cache (keeping in mind you need to hold onto it until the client ACKs it), and CPU cycles (probably still a decent chunk of overhead even with AES-NI).

They are completely different use cases though, so it shouldn't be too surprising that the results are different.

Gmail: Lots of small requests which each last a short time.

Netflix: Small number of large requests with connections that stay open a long time (possibly hours).

I think the context here is that Netflix does not want to serve video over HTTPS but some browsers are talking about restricting EME DRM so that only HTTPS origins can use it.

Without encrypting the video contents, anyone sniffing the data stream can easily tell what you're watching, when you paused, or skipped, or re-played, etc. They may not be able to steal your money with that data, but it still exposes real, personal information.

(though... I was just thinking about this, and due to the use of variable-bitrate encoding, the varied rate of data transmission over time (even though encrypted) would give each video a fingerprint that an observer could use to determine whatever it is that you're watching, anyway, albeit with greater effort that simply inspecting the unencrypted stream)

HTTP can be used not only for spying on you, but for attacking you with malware as well (or even something as "innocent" as being injected with the ISP's own ads):

https://citizenlab.org/2014/08/cat-video-and-the-death-of-cl...

You'd be surprised how many ISPs want to MITM media.

It's not about encrypting video files specifically, but:

* Not leaking any tracking or user-identifiable information. EME DRM plug-ins track users, but browsers don't have control over this (and I guess it may even be illegal to audit EME DRM security due to DMCA), so it's best to have it on HTTPS just in case.

* All new browser features require HTTPS (and EME DRM plug-ins are new-ish), because browser vendors want to deprecate HTTP, and don't want headache of mixed content or HTTPS alternatives built on DRM.

In the HTML5 context, adaptive streaming is done with the MSE API. This involves fetching the media data using XHR. Since mixed-content XHR is blocked, if the app is on an https origin, the media data has to come from https origins, because cross-scheme CORS is blocked for security reasons.

You want to have the app on https for several reason, starting with the security of session cookies, but in the case of DRM, there's a reason for browsers to seek to limit the EME API (the JS API for communicating with the DRM component) to https (though AFAIK no browser does so at the moment):

DRM schemes usually implement node locking, i.e. anti-cloneability of the DRM state from one device to another. The obvious way to do this involves keys that no other device has. If information about the possession of a persistent key that no other device has is exposed to a network eavesdropper (e.g. by presenting, in the clear, a certificate for a unique key), the eavesdropper can use that piece of information to track devices and, therefore, users.

While this problem could be mitigated in the design of the DRM protocol, to move addressing the problem out of what the DRM component controls to what the browser controls, limiting EME to https is the most straight-forward solution that can be implemented on the browser level. And as noted above, this has the side effect of limiting the MSE-feeding XHR to https, which means the media data needs to be served over https.

They do not use AWS for streaming, just for the control plane.

I was at the talk and they did not mention hardware offload, but there are a lot of problems with having full control.