Hacker News Reader

Comment by drewg123 | Hacker News Reader

We've been running kTLS + SSL sendfile on FreeBSD at Netflix for the last 6 or 7 years. (We had local patches to nginx, before nginx did them "right", and 2 versions of kTLS before the 2nd version was upstreamed to FreeBSD). The savings in terms of CPU use and memory BW are pretty substantial. Especially when you use a NIC which can do in-line kTLS offload, then things basically go back to pre-TLS costs because the buffers are not touched at all by the CPU.

BTW, FreeBSD 14 supports cha-cha poly. But is far more CPU intensive than GCM, so I'd advise against using it.

View on HN

Was CPU a bottleneck for your throughput, or did you just want some more headroom on those machines?

I've been in the room with some web scale companies, but never was aware of any big excess TLS termination costs, but maybe I just didn't ask and they had some specialized hardware/software that I didn't know about.

I'm not at Netflix, just summarizing. Check out the Netflix presentations like [1]; at 400gbps, the bottleneck is RAM and NUMA fabric. Doing bulk crypto on the CPU requires double the RAM bandwidth, and that kills the throughput regardless of the cost of cipher computation.

TLS termination costs are a large factor for Netflix because their CDN appliances want to have as much throughput as possible in a small space. At 'web scale', it wouldn't be a big deal to be happy with 100gbps througput and use 4 boxes; but getting ISPs to install more boxes is hard, it's better to get one box to work as fast as possible.

Personally, I've seen it be an issue; dual xeon 2690 v1 or maybe v2 couldn't push 10G via TLS, but could via plain text. But the v3 chips had better acceleration and could do 2x10G no problem. I never got access to more than 2x10G networking, so no idea what the limit was. That was under managed hosting (dedicated bare metal), so no ability to do networking beyond what was offered. At Facebook, they tended to use smaller servers and more of them, and I didn't deal with TLS as much.

[1] https://people.freebsd.org/~gallatin/talks/euro2021.pdf

The limit for FreeBSD software kTLS with our workload is a little over 100Gb/s (using a 2x100G NIC) on a 2697a v4. I think most people only get 25-30Gb/s out of the same CPU when not using kTLS.

The bottleneck is both CPU and memory bandwidth. Doing kTLS in software (or in a lookaside accelerator, like QAT) doubles memory bandwidth requirements because something (CPU or accelerator hardware) needs to read the buffer to be encrypted, and write the result into a new buffer.

And doing TLS in userspace, rather than kTLS is even worse, because it disables sendfile. That means you now have extra copies across the user/kernel boundary.

Not at netflix but at scale any savings in CPU usage translate directly into spend savings. If you have 100 machines at peak and autoscaling rules in place, you can generally assume that a 1% saving in CPU usage translates into one less machine running. You never get "headroom", you just have less machines running at any given time.

That's only true if CPU is the bottleneck, which is why I asked. I would expect network throughput to be the bottleneck, but maybe that's wrong nowadays with cheap 200gbit nics.

Video streaming is a bit different from "normal" TLS web serving. For video streaming the network is much more likely to be the bottleneck, but traditional TLS web serving tends to send a lot of fairly small packets, and saturating even 40gbe with small packets of TLS is non-trivial.

My experience is that at small (think less than 700 bytes or so) packet sizes, TLS overhead can be a very real cost as you scale up, and it's easy to hit "walls" where really you just need to buy more servers because the engineering cost of getting your servers to really saturate their NICs isn't worth it. How much of a big deal that is for you will depend a lot on exactly what sort of a thing you're serving, though.

I'm curious why TLS packets are small in your experience. We use 16k TLS records, and send at whatever MSS the client has negotatied with us. Thats normally something "normal", like 1420-1460 bytes. Plus we use TSO, so the NIC sees large sends of up to 64KB.

Random question: Do you force Netflix clients onto the ciphers which are most efficient for the Netflix servers, or are there cases (I'm thinking mobile devices particularly) where it makes sense to use the ciphers which are most efficient for the clients?

Pretty sure the best choice is basically always AES GCM because most modern chipsets can hardware offload that. Curious to hear the answer, though.

Genuine quesion: In 2022, is that still dependent on "Can I trust my (HW?)random number generator"?

I don’t believe so. AES is symmetric-key encryption; if you do it incorrectly, it doesn’t decrypt at all. The only place you can mess up, then, is in the mode, which I don’t believe is accelerated by most (any?) AES encryption instruction sets; instead, they mostly handle doing an individual round of encryption/decryption.

Public key crypto systems seem like they would be much scarier to have hardware acceleration, though I’m sure if you broke it down to low level enough bits you could make it impossible for the hardware to “break” it (aside from, well, if it decided to just maliciously subsitute your code for its own. But it could do that without extension ISAs.)

GCM is typically implemented in Intel assembler with the PCLMULQDQ instruction

https://www.intel.com/content/dam/www/public/us/en/documents...

How is the story any different for public key crypto?

I am not a cryptography expert, so please take what I say with a grain of nacl. The parameters you choose for public key crypto systems can weaken the cryptographic properties of the result. For example, nonce reuse can single-handedly destroy your security. Similarly, with RSA, padding is crucial to security. Even if you write the extension ISAs such that they never choose parameters, there’s still probably some room for their to be broken output that seems to work.

It of course depends on the random number generator. That is how you create the keys and sometimes the initialization vector. But AES itself is deterministic so it’s not possible to backdoor that part.

x86 and ARMv8 do, but not ARMv7

And one huge exception - RPi CPU.

There are cases where it makes sense, but I'm not sure that mobile devices that are likely to be playing video is it. Chances are, they'll be burning more power on the screen backlight than the CPU to do AES vs something else (assuming they're not accelerated for AES). There's two sides to the argument, but easing the burden on servers lets one server serve more clients; it's easier to justify 2x the cost for crypto on the client than on the server, because most clients aren't bottlenecked on crypto and some servers are. (Of course, I'm usually a server engineer, so of course I want my servers to have less work ;)

Choosing ciphers for ease of the client makes more sense, IMHO, when the client is really constrained, like a feature phone or tiny IoT things.

Reply on news.ycombinator.com