Netflix now supports TLS 1.3
netflixtechblog.com
netflixtechblog.com
As I understand it, they use HTTPS to prevent spying and data-mining by unscrupulous ISPs. It doesn't affect their DRM at all, which would work just as well over plain HTTP.
There was a skit I saw a couple decades ago where a person was showing off his radar detector, then in order to combat that, the police had developed a radar detector reflector, so then he had made a radar detector reflector protector or something like that, then they made a protector detector, and so on. It was a couple minutes of explaining his best efforts to counter the police counters to his counters and on and on. But, you know, funny.
The trace busta busta busta.
Creating fast.com always seemed like a pretty brilliant move by Netflix to me.
Almost nothing supports ESNI yet. Chrome does not have it yet. Firefox does but it very difficult to enable, there is a config flag but it does nothing on its own unless you also enable DNS over HTTP in Firefox.
OpenSSL has no support for ESNI yet either.
ESNI also never tells the user if it is working or not yet, making downgrades fairly easy.
ESNI is a long way from being deployed, let alone useful.
Users can test their ESNI support online here: https://www.cloudflare.com/ssl/encrypted-sni/
But, I encourage you to try to get it working in _your_ firefox, where that site says you are in fact using ESNI. It is trickier than just enabling a flag.
If you think this means people are using it, sure. Firefox has ~9% of market share, and a very small percent (maybe 0.1%) of those users have correctly configured this, and when those users visit a page hosted by cloudflare they will may be use ESNI. That is in fact more than zero users. It is a very very small number today, but hopefully it will grow in the future.
And while that page you link tests ESNI, I cannot verify that any other domain supports ESNI. I am unaware of any tooling that lets me test that today.
Only if you use their built-in DoH resolver. The public keys for ESNI are distributed by DNS records and as I understand it there's a bunch of work to be done for Firefox to retrieve these records from classic DNS servers. That work has been classified P5 (we won't do it, but might accept patches)[0].
Does it work at all though? I does prevent paying customers from getting higher quality streams. But aren't all the shows still widely available in torrents at whatever quality you want?
> From the field test, we are confident that TLS 1.3 provides us a better streaming experience.
so they actually measured an improvement in performance metrics.
Or do you just not think that more security is a good thing generally and should be done for its own sake?
I'm all for HTTPS everywhere, but for Netflix and YouTube, that means encrypting exabytes of data. That's going to cost them quite a bit. Seems reasonable to go for more detail than It's good practice in general.
> Are you hinting at something here
No, as I put in my first comment, I think we know the specific reasons (untrustworthy ISPs), it's just interesting they didn't mention them.
[0] https://whydoesaptnotusehttps.com/
[1] https://developer.valvesoftware.com/wiki/SteamPipe#Server_ad...
It seems like the benefits of transmitting your streaming securely (e.g. defeating interloping ISPs) more than exceed the costs, hence why they're doing it.
When people whine about the overheads of HTTPS, they're generally talking nonsense, as the overheads are typically marginal. Video streaming providers are the exception.
https://arstechnica.com/information-technology/2015/04/it-wa...
apt used to be 100% HTTP based. I believe they've changed course recently though. Unsure about Steam.
Steam were clear that the reason they adopted insecure HTTP (previously they used a proprietary protocol) was to enable web caching, by ISPs and other organisations. I can imagine that being very beneficial at a commercial LAN party, for instance.
Related reading: https://wiki.debian.org/SecureApt
Because secure should be the default where we can do it.
If I'm watching a show, I don't necessarily want every single router along the way knowing what show i'm watching. Either because of advertising profiles or just because I don't want people knowing that I'm binge watching a specific show, or to go to an extreme, perhaps because what I'm watching isn't legal where I live.
If I'm watching a documentary or news show, I don't want the possibility that someone could modify my stream and change parts of it. Misinformation is a big deal, and if we have a simple solution to get rid of whole attack vectors (especially before they become common), why not use it?
And TLS gets rid of other attacks as well. token stealing is a lot harder if EVERYTHING is TLS encrypted vs if only some things are. It can also secure against vulnerabilities in the software that reads the streams. They no longer have to harden their systems as much against potentially hostile data, they can be behind well tested TLS termination that only allow verified data through to the "backend".
Everything should be encrypted in transit, zero exceptions. Just because you don't think it's important to hide doesn't mean that others don't. Just because you think your parsing is secure doesn't mean that another layer isn't useful for defense in depth. We have the technology. It's free, cheap to run, and doesn't hurt the user experience in any significant way, so why not use it?
Unless Netflix has also started padding their compressed video segments to a deterministic size, TLS doesn't really help with the "I don't want people to know what video I'm watching" problem, as the encrypted media segments are first compressed using a variable bitrate encoding that causes a unique pattern in their resulting chunk sizes; and so, with a little database of signatures--apparently even just at the TCP level, without caring at all about the HTTP and TLS layers... which surprised even me, and I'm quite cynical about this stuff--you can pretty rapidly fingerprint what movie someone is watching from a passive traffic dump.
https://dl.acm.org/doi/10.1145/3029806.3029821
> Identifying HTTPS-Protected Netflix Videos in Real-Time (2017)
> After more than a year of research and development, Netflix recently upgraded their infrastructure to provide HTTPS encryption of video streams in order to protect the privacy of their viewers. Despite this upgrade, we demonstrate that it is possible to accurately identify Netflix videos from passive traffic capture in real-time with very limited hardware requirements. Specifically, we developed a system that can report the Netflix video being delivered by a TCP connection using only the information provided by TCP/IP headers.
> To support our analysis, we created a fingerprint database comprised of 42,027 Netflix videos. Given this collection of fingerprints, we show that our system can differentiate between videos with greater than 99.99% accuracy. Moreover, when tested against 200 random 20-minute video streams, our system identified 99.5% of the videos with the majority of the identifications occurring less than two and a half minutes into the video stream.
Note that this exact same kind of attack has been applied successfully to other places where a pattern of sizes can differentiate usage, including: Google Maps tiles (which are either variably-compressed fixed-dimension images or variably-dense fixed-region vectors), web pages (which are different sizes already but also link to images and other resources that are random sizes, which helps create the fingerprint), and non-personalized type-head search queries (where the result for each letter brings up some unknown set of titles and URLs; as you type each character the sequence of sizes for your intermediate results can expose the query, potentially even without needing any model of the likely keywords).
Maps: https://ioactive.com/ssl-traffic-analysis-on-google-maps/
Search: https://eprint.iacr.org/2014/959.pdf
(edit: I just realized it might be interesting for me to note that I am the lead developer of a new VPN protocol called Orchid and have "attempting to solve as many of these issues as generically and yet with as little required overhead as I can" on my todo list; to be very clear: I don't handle any of this stuff yet at all, but should get to it in the next few months... while the company behind Orchid that is paying me to work on the protocol talks a lot about itself already, I'm really intending to do my own "launch" of the product once I solve some of the stability issues and finish my traffic analysis mitigations.)
But in the end it still doesn't mean TLS isn't worth it for all the other benefits (not that you implied that).
The issue then is that no matter what you do with the compression algorithms, the user is downloading these chunks and you can see the pattern of packets in one direction setting up requests and the packets in the other direction replying with the chunks and you can fingerprint what movie someone is watching. It isn't some kind of encoding issue, as the data is all encrypted: it is the entire concept of taking fixed length segments of a movie that will compress to some non-determinstic size. If you take the first two minutes of every Star Wars movie, divide each up into 10 second segments, and then compress those segments, the sequence of sizes of the segments will be pretty unique.
What is so great about this particular paper is that you don't even really need to analyze the TLS layer and try to pay close attention to really figure out the request/responses: they just fingerprint the TCP flow and that's sufficient, which in retrospect doesn't surprise me as what you are really looking for is some kind of rate of requests to responses for the chunks over the course of those first few minutes of watching the video, and don't really need to know for sure where the boundaries are: you have a long enough sequence and a small enough catalog (there aren't tens of millions or billions of videos on Netflix) to get a really strong fingerprint using just the relative rates.
To fix this you really need to either inject extra random traffic (such as extra packets to the server that break up the request rates) that adds so much noise that you can't figure out the signal "in time"--if it takes longer to fingerprint a movie than the length of a typical movie, that's "good enough"--or you need to destroy signal (which is a better description of what we do if the video segments are all the same size: at that point all movies are by definition the same sequence over and over again; if you then pad the length of every movie to the same 4 hour runtime and force the user to download all of the padded black video frames, you essentially 100% solve the problem ;P).
That said, as noted by someone else on this thread, there has been some work done figuring out what people are saying by analyzing encrypted speech packets; but I imagine that kind of technique would be almost impossible to pull off with these segments on the order of multiple seconds long, and including the video in the stream would seem to make that a total non-starter.
Every Netflix client that is watching say 1080p Rick & Morty s1e2 fetches an index of the bits that make up that video, and then fetches the relevant chunks of data.
So if you just fetch that index, no need to "watch" the video, you know OK 1390450 bytes is the 1st chunk of data making up that video, and then 2046917 bytes is the 2nd chunk. When you see that somebody fetched 1390450 bytes of encrypted data from Netflix, they're probably watching s1e2 of Rick & Morty in 1080p. When they fetch 2046917 bytes of encrypted data that confirms it.
So it's "non-novel" but they don't need to (relatively expensively) actually watch the videos they want to match. It's likely any sophisticated adversary has an index of all the common videos from sites like Netflix, the Play Store, Disney+ and so on.
There is definitely more a client could do to make this harder, but there'd be a sharp trade-off where to make it impossible for passive eavesdroppers to know what you saw costs both you and the "broadcaster" a lot of bandwidth, and bandwidth ain't free.
I'd like to see Netflix at least gently stroll in that direction, maybe use some TLS padding to make it a bit harder to do this trick, but I don't expect them to set off at a sprint and ruin their profitability in pursuit of a non-goal.
Off the top of my head: if you took a live streaming approach along the lines of live TV, or indeed Google Stadia/GeForce Now/OnLive, then you could presumably make the traffic patterns invariant over what was being watched. You'd need to ensure the bitrate didn't drop during low-detail scenes, I suppose, similar to what you mentioned re. Skype below.
That approach would of course be far more intensive on server resources, but I'm sure a similar level of privacy could be achieved without such inefficiency.
for a single user it's possible. for a network with gigabytes of data, it's basically impossible. it's not just a "tcpdump" and compare against fingerprint.
btw. it's also outdated.
also eme basically solves this on the audo and video side. (maybe not the best solution tough)
the attack inside the pdf does basically mention the old technology which netflix used and not dash+eme inside video element (MPEG-CENC). i.e. widevine which is kinda funny because the browser drm is highly controversial, but in this case serves as additional privacy since video metadata and content get's encrypted, no matter which transport layer is used.
EME encrypts the video content, yes, but I don't see that this changes things. Netflix uses HTTPS (TLS) encryption over the top of the EME encryption, as this thread's title states.
> even metadata is encrypted.
I don't believe so. If you're delivering EME-encrypted blobs over insecure HTTP, an ISP will be able to see which blobs you are requesting, simply by their URL.
Aside: I recall reading that Netflix's CDN servers ('OCAs') store EME-encrypted blobs, so only the HTTPS encryption burdens the delivery server's CPU. Unsurprising, of course.
> without eme you can basically extract the video via tcpdump
Six years ago, sure. Today, no, as the stream is sent over HTTPS.
> in this case serves as additional privacy since video metadata and content get's encrypted, no matter which transport layer is used.
Perhaps unencrypted streams are still used to support legacy devices, but Netflix are committed to maximal use of HTTPS.
I don't see how EME's additional layer of encryption changes anything as far as privacy and unscrupulous ISPs are concerned.
Also you don't need a precise measurement. Just measuring a random 10k users in 1M network daily will give you more precise viewership measurement than anything Netflix publishes.
edit - said Skype initially but not sure it was specifically skype
Unfortunately for video CBR is generally thought to be too expensive to be acceptable. 50kbps CBR Opus wastes 22Mbytes of data in an hour if you're silent compared to a good VBR codec. A small price to pay for improved security. But a CBR Netflix encoding might move several gigabytes per hour of extra data compared to a visually similar VBR encoding, and that's going to really hurt both Netflix and quota restricted customers.
The TLS 1.3 on-wire format is weird because it needs to pass rusted-in-place TLS 1.2 only middleboxes. One of the convenient side effects is that you can add one or more bytes of padding "free". Any 1499 byte packet can be transformed into a 1500 byte packet just by adding one byte of padding whereas many naive formats couldn't do that.
So this means you could "right size" packets just before sending, so that you still send the same number of packets but they're always full, reducing the ability to distinguish their contents by length.
Signal does something more sophisticated to defeat this sort of analysis for its GIF support. The local client does two or more overlapping GETs. So maybe it needs a 14230 byte GIF but it fetches 8430 and 9401 bytes of data then throws some of it away to re-assemble the 14230 bytes needed, an adversary is left with too many possibilities. But it's probably a bit much to expect Netflix subscribers to pay extra for bandwidth just to defeat an adversary who wants to know how much of Tiger King they've seen.
Pervasive Monitoring Is an Attack
There are only a few tens of thousands of shows on Netflix, so it ought to be easy to identify.
One way to defeat it, that is already happening to some extent (though not for this reason), is buffering. Maintain a large enough local buffer, and fill it at a constant bitrate, rather than runrate, and you lose this attack vector, no?
Adapting the research to other non-CBR streams doesn't sound far-fetched.
However, I strongly suspect that Netflix's video and audio are sent in different streams. Occasionally video for some titles is missing but audio gets through, confusing our daughter to no end. So while you can't infer individual syllables from the audio stream (as you would from VoIP), the audio streams should have varying enough size and chunking characteristics that allow to identify them.
Interesting data point on the separate AV streams. I can't say I've noticed that myself.
Presumably it depends on the device, but I vaguely recall testing this out on a Chromecast, and I found Netflix to buffer up for several minutes. Some other streaming providers, such as Rakuten, only buffered around 15 seconds.
> I strongly suspect that Netflix's video and audio are sent in different streams
I think this might be right. They presumably want to support lots of different combinations of audio codecs and video codecs, so keeping them separate would make sense. Pure guesswork on my part though.
1. Integrity is redundant with authentication, so really you could say they're ensuring confidentiality + authentication. You can't authenticate a thing without implicitly obtaining assurance of integrity. It's a strictly stronger property.
2. Confidentiality is (usually) insecure and unreliable without authentication. Without authentication you have no PKI for a key exchange to symmetric encryption, so you can't even do TLS in the first place. And if you don't have a carefully applied MAC or a native AEAD mode, your symmetric mode isn't that secure either.
So really what you're asking reduces to the question of why they need the most sophisticated TLS scheme for encrypting their streams. If they want the most secure TLS scheme for confidentiality, TLS 1.3 is the way to do it. They explained one particular facet of why this is the case, re: perfect forward secrecy.
The encrypted media extensions (EME) are only available on secure contexts, which means the page needs to be HTTPS. Segments are typically fetched with standard XHR/fetch, which means they need to use HTTPS on secure origins as well. You really don't want to have to deal with mixed content.
Unrelated to DRM, Apple's ATS on iOS kinda requires developers to only use HTTPS origins in their apps.
At that point, you might as well just do HTTPS everywhere.
I didn't know that, but you're right. [0] There was a time Netflix didn't use HTTPS. [1] I guess they must not have supported HTML5 at that time, and must have been using Silverlight for in-browser viewing.
[0] https://www.w3.org/TR/encrypted-media/#privacy-secureorigin
[1] https://arstechnica.com/information-technology/2015/04/it-wa...
I definitely have had to enter payment details on Netflix.com before
I can't speak for all of Netflix's motivations, but one of the biggest for OTT platforms is preventing MitM of your streams by middle-proxies - including ISPs.
That MitM prevention encompasses data mining/user privacy, ad injection/replacement, and maintaining control over stream quality (Quality of Experience).
There used to be [and still is, in some places] a lot of transcode-to-lower-bitrate-and-cache-inside-our-network behaviour, especially amongst mobile carriers. Reduced stream quality would reflect on Netflix, not the ISP who might be doing this transparently. A user switching up to 1080p or expecting 4K HDR (as appropriate) wants to get that, and Netflix [or others] want to deliver that experience as intended.
The reason we see a LOT less of this now is due to how easy (and thus, widespsread) TLS became.
Joking aside, Netflix's web page runs on a vast variety of devices from phones to smart TV's which don't have the same security profiles. You don't want someone to be able to inject a packet to your Netflix stream to pwn your TV.
I run an out-of-band network tap and have seen only TLS 1.3 traffic for some time now.
At least, when I had a computer configuration that they said was unsupported for HD due to DRM reasons, I was still able to get Amazon's own videos in 1080p
Have the anti privacy crowd come out in droves now that we have a public desire for contact tracing & there's a desire to scapegoat why netflix et al have reduced stream quality due to increased load?
& HTTP is not an option. I for one enjoy my ISP not being able to inject ads into my streams
Netflix already had a secure system with their original business model of mailing you the content.
Finally, allowing advertisers to profile us based on our browsing data is one step before allowing the police to do so as well. Maybe it's the same step.
https://news.ycombinator.com/item?id=22933774
https://news.ycombinator.com/item?id=22934040
https://news.ycombinator.com/item?id=22934379
Maybe these comments appear first because they're less likely to've read through the article. Otherwise it'd have to be something to do with the demographic of people watching /newest. But it was posted 4 hours before I made my initial post, so I don't know
My guess is some people have a greater hatred for DRM, which makes Netflix "bad", and so anything Netflix does security-wise that isn't removing DRM becomes irrelevant.
But those comments seem to be mostly downvoted now anyways, so they don't seem to be representative.
A few things are whitelisted but the handful of things that break aren't necessary anyway.
OCSP is to query random CA web servers when opening a SSL connection to verify if the certificate in use was revoked. It's broken by design because servers simply don't have internet access.
Microsoft Windows, for example, allows administrators to re-publish external CRLs internally, but this only works for the built-in crypto API. Non-native applications generally won't pick this up, including most command-line tools. Blocking outbound port 80 is a mistake.
A colleague said it best: Certificate private keys are like your password. Tell no one, ever. The public key is like your passport. It's safe to hand to a border officer. CRLs are the lists of passport numbers used by known terrorists wanted by Interpol. Preventing the border officer seeing that list just lets the terrorists through the airport. This is never good.
Similarly, most firewalls ship with "drop" as the default rule, which means that a small error in the firewall rules will just end up with 20-120 second timeouts throughout the network. If instead they "rejected" flows with an explicit deny response, this failure mode would be instant and easier to diagnose.
Just yesterday I was trying to diagnose a packet loss issue on a network where some security troll blocked all ICMP between all endpoints. The last time "ping" was used as an attack vector was in the last century, but apparently it's important to prevent any hope of diagnosing entire categories of network issues. Unsurprisingly, their network has atrocious performance, random periods of high packet loss, and asymmetric routing issues. This will not get resolved until the idiot in the security team quits or retires.
PS: I suspect their issue is large UDP packets getting dropped on some paths due to MTU issues. To diagnose this without ICMP in a complex network is basically impossible. The automatic fix is to enable Path MTU Discovery... which requires ICMP!
Every large environment I've been to that has these security trolls in charge with nobody to reign them in was a shitshow of broken systems performing catastrophically badly because they spend 99% of the time waiting for timeouts.
PPS: Why firewalls don't simply have a default, auto-updating rule for "Enable CRL & OCSP outbound" for all known public CAs is beyond me. This is an extremely common requirement, to the point of being practically mandatory for any medium-large enterprise. If you firewall doesn't have this feature, and about a dozen similar ones, you're getting ripped off. You're probably paying some decent fraction of a million dollars for a bag a tools, not a complete solution.
Specifically, if you use HTTPS for OCSP then now you need OCSP for the certificate for the HTTPS connection you're doing to get OCSP, so you open another HTTPS connection, and you need OCSP for that certificate, and then you open another HTTPS connection and...
But most browsers don't do OCSP by default because realistically you need fail-open behaviour and if you have fail-open behaviour there's no meaningful security benefit so why bother?
The top answer is a stackoverflow answer saying no way. "Putting SSL everywhere is only a way to get a warm fuzzy feeling on security that is no good. It [SSL] is usually only used as a distraction allowing administrators to disregard actual security issues."
https://stackoverflow.com/questions/2177159/should-all-sites...
OT, but why am I still paying full price for less-than-full quality? Same with Amazon - why am I paying for two-day Prime shipping even though it's impossible to get that level of service right now? /rant
You can always cancel your contract with them if you don't like it.
Note: I'm not even sure they have disabled 4K, it's just an example.
This is not quite true - it's less about _Netflix's_ load and more about load on the _access networks_ (end-user ISPs).
They are seeing longer peaks, increased congestion (fewer folks are leaving their homes), and thus the reduction in default streaming quality by many streaming platforms was a move to limit congestion on these last-mile networks (and some IXs) where possible.
The EU & India anticipated the most congestion here, which is why you likely saw these announcements start with the EU before becoming global, as other ISPs/IXs/govts began to think about the same.
Disclosure: I work for Google in this space, but don't represent them here.
Comment threads don't represent the community view. They self-select for commenters who object to something. This is particularly true early on, since objecting is usually a reflex reaction that leads to quick comments. Reflective comments, the better kind, take longer to appear—because reflection is a slower cognitive process and because they take longer to write. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
The third wave is people looking around and not seeing the second-wave description of the thread really matching the comments (we got one of those too: https://news.ycombinator.com/item?id=22935245). That's because the original objections have since been supplemented by more interesting comments that were slower to roll in. It may also be because the janitors a.k.a. moderators have been active in the thread, sweeping up and so on.
Edit: Speaking of janitorial work, I've downweighted this subthread now, which is one of the things we do. Check out the effect: at least as of this moment, if you scroll back to the top, notice how much more enticing and curious the conversation is, than if this were the first thing were seeing. It's as if the thread can breathe.
What's the better way to play out this dynamic? Since there were three comments I didn't think it'd be optimal to spam by replying to all three individually. Maybe picking one to reply to would be better. I don't think downvoting is the right play, but ignoring may be
Maybe responding with agreement to the article, without objection to the objection
The trouble with an indignant anti-crap comment is that it can't help but repeat the crap it's objecting to, because that's what it's about. Negation is a form of repetition. Crap subthreads tend eventually to get moderated by users, but that doesn't happen with their anti-crap counterparts. Usually those get upvoted to the top, since people love defending the unfairly criticized and we all respond to indignation. But then it sits there, choking out new discussion with metafumes.
The best way to counter bad comments is to post a good one—for example, a reflective comment about the specifics of the article, or a curious comment about something unexpected in it. This isn't always an option, though, because one doesn't always have something like that to say. I think responding to reflexive negativity with a pleasantly-worded objection can still be helpful—just not as a top-level comment.
As an example, the NVIDIA driver download pages switch to http:// for the actual downloads although the rest of the site is HTTPS...
It's interesting to see which companies have difficulty keeping up.
Switching from TCP to TCP over UDP in order to run congestion control (and other things, such as segmentation) in userspace instead of kernel space makes more sense if you don't control the kernel; but Netflix controls the kernel on the content appliances. Netflix has also put a lot of effort into moving more of the processing into the kernel to avoid copying data and context switches between the kernel and userspace; moving congestion control into userspace would most likely have a large negative impact on their serving bandwidth per appliance.
If they support TLS at all, why not 1.3? Do they have some dodgy hack that avoids a performance cost in 1.3 that is not in 1.0? Or is it just sloth?