Optimizing HTTPS performance | Hacker News Reader

Optimizing HTTPS performance

istlsfastyet.com

150 pointssshravan38 comments

Good points. The author focuses on (a) have server do everything itself or (b) use SSL accelerator hardware. There's a middle ground which has other benefits depending on your use-case: I/O offloading hardware that also handles SSL. These are often PCI networking cards that handle the packet processing, firewall, IDS, maybe management, SSL/VPN, etc. The host CPU works at a high level where it's only interrupted upon significant events or with most work done. The load on an I/O-bound server is so small that cheap, embedded CPU's can handle it. A CPU-bound server gets a great boost in performance for not worrying about massive I/O load.

So, there's still potential with that model. Use host CPU if that's all you need, though. Just remember that a few $200-300 networking cards might be a better deal than whatever HSM vendors charge these days.

I'm currently adding https (TLS) support to my blog, using LibTLS (part of LibreSSL).

My initial impressions are that TLS handshake is very computationally expensive.

Stress testing my webserver with plain HTTP connections:

7980.2744 queries/s

With HTTPS (TLS) connections:

23.059916 queries/s

The slowdown is due to high CPU usage.

I certainly wouldn't classify that as fast!

I've only done a bit of work on this so it's possible I'm misusing the LibTLS API however.

The site and the linked resources talk about handshakes: only a small percentage of your traffic should actually require a full handshake, since most of your users are going to request multiple files. Basic benchmarks don't do that and therefore provoke a worst-case you are unlikely to see in real usage. Might be relevant for APIs, ad servers or similar though.

How did you measure this?

When measuring on one of my servers that exposes both HTTP and HTTPS, there doesn't appear to be any significant difference.

HTTP:

  % wrk http://hostname
  Running 10s test @ http://hostname
    2 threads and 10 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency     5.11ms    5.42ms  62.85ms   96.08%
      Req/Sec     1.10k   224.03     1.31k    87.04%
    17703 requests in 10.01s, 6.89MB read
  Requests/sec:   1768.57
  Transfer/sec:    704.58KB

HTTPS:

  % wrk https://hostname
  Running 10s test @ https://hostname
    2 threads and 10 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency     4.75ms    4.09ms  50.37ms   96.48%
      Req/Sec     1.10k   228.86     1.34k    79.35%
    17014 requests in 10.01s, 8.39MB read
  Requests/sec:   1700.32
  Transfer/sec:    858.38KB

I tested it with some of my own code, 8 threads making requests to my server at localhost.

Does this program 'wrk' re-establish a connection for each request, or is it re-using them?

wrk[1] does appear to reuse a connection for each request.

[1] https://github.com/wg/wrk

Any idea how HTTPS manages to have a significantly higher throughput? At first blush this doesn't quite pass the sniff test - it looks like there's another factor in play here.

That's probably because the response to this particular https URL is larger; http on this server simply redirects to https and is therefore a smaller response.

Not that this matters here; the top level comment measured requests per second, not bytes per second. If anything, the smaller response for http would give http an advantage in this comparison.

What key size and cipher suite are you using? (If you're willing to post the URL, we can have a much more informed discussion, but a sanitized copy of an SSL Labs test or even of `openssl s_client -connect localhost:443` is also workable.)

Also, what hardware are you using? This site is aimed more at companies who fear of losing business because their site is slow, than people hosting a blog on a home server; if you don't have hardware-accelerated crypto (like most server-class machines have, whether or not you explicitly looked for it) its argument is less valid.

I'm using all the defaults, as provided by LibTLS, so I'm not sure. I'll have a poke round and tell you later.

The TLS support is not deployed to the internet so posting a URL wouldn't be very helpful.

Hardware is my Ivy bridge i7 with 4 physical cores.

Yeah, Ivy Bridge is new enough. Are you negotiating an AES-GCM suite? Also, are you using a 2048-bit key if you're doing RSA? (A larger key almost certainly provides no more security, since your intermediate is almost certainly 2048-bit, and will noticeably slow down handshakes.)

LibTLS tells me what was negotiated was 'ECDHE-RSA-CHACHA20-POLY1305'

Edit: some more output:

  New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
  Server public key is 1024 bit
  Secure Renegotiation IS supported
  Compression: NONE
  Expansion: NONE
  No ALPN negotiated
  SSL-Session:
      Protocol  : TLSv1.2
      Cipher    : ECDHE-RSA-CHACHA20-POLY1305

Yep. Keep-alive is your friend. If you use Apache or similar web server where keep alive means using a lot of RAM, you can place HAProxy or similar load balancer in front of it. HAProxy keeps the connections with browser alive, while connections between HAProxy and backend web server are closed and re-established all the time. Of course, this means that SSL is terminated on the load balancer and not the webserver.

One problem is "TLS" isn't just one thing to understand and test against. TLS involves dozens of optional cypher suites and each one has a unique performance profile.

Just for fun, add !kEDH to your server's list of allowed cyphers then see how your performance changes.

Another performance impact some people overlook: your key size is also computationally expensive. There's not much reason to have a key larger than 2048 bits, so if you made a 4096 bit key just for fun, roll it back to a 2048 bit key for better performance too (plus, your mobile viewers will thank you for not burning their battery life needlessly).

Stupid question:

A lot of HTTPS's value comes from authentication, not encryption. Has anyone investigated using something like the TLS_RSA_WITH_NULL_SHA cipher suite for HTTPS, to provide auth and integrity without the computational cost of symmetric encryption? Would that even provide a meaningful performance benefit?

I'm guessing that it's disabled by default in most browsers, and figuring out a sane user experience to allow users to differentiate between encrypted and non-encrypted secure traffic seems like a very hard problem.

A similar idea is to allow pages to download static content over HTTP if they include the hashes of the content. Things like CSS and javascript need authentication so you know they haven't been tampered with, but don't necessarily require encryption. (The authentication here is from the fact you would download the page containing the hashes via HTTPS). Using this to download the global static content for a site doesn't even leak which individual page you were looking at.

On the other hand, it complicates things a lot without really giving much advantages. The only time I can think of where it might be useful is for using CDNs from a HTTPS site, without having to set up HTTPS with your CDN.

If you're concerned about things getting tampered with, encryption guarantees that it won't. With your suggestion of hashes + HTTP, if it is tampered with, the page silently won't work as intended. With hashes + HTTPS, loud errors happen when things are tampered with. And because it's static content, deliver over HTTPS once and cache the hell out of it.

> If you're concerned about things getting tampered with, encryption guarantees that it won't

Sure, HTTPS protects against tampering, but its not the only way. The parent comment seemed interested in ways of solving the same problem without the CPU overhead of encryption (whether this is a valid concern or not is a separate question).

In fact, hashes offer superior protection when loading resources via a CDN since you don't need to trust that the CDN won't tamper with things.

> With your suggestion of hashes + HTTP, if it is tampered with, the page silently won't work as intended.

Why? This would require browser support and it could whine about mismatched hashes as much as it wants.

---

I'm not seriously proposing that we should do this, but I find it interesting to think about the different ways that things can be done.

The performance benefit would be hardly measurable. First of all the performance impact of HTTPS in general is almost irrelevant. Second, even if you care about that irrelevant performance impact, unless you transmit really large files the biggest hit is from the asymmetric part. You'd still have that with a NULL cipher.

You would also introduce a significant risk. How would you communicate your "authenticated, but not encrypted" connection? Looks like HTTPS? Like HTTP? Something new? Confusion for the user (who doesn't know the finer details between authenticity and secrecy) is guaranteed.

In the end you'd introduce a less secure variant of HTTPS without any significant advantage. And you add complexity and the risk that it gets used in situations where secrecy matters, but the people responsible didn't properly think about it. Doesn't sound like a good idea to me.

"First of all the performance impact of HTTPS in general is almost irrelevant."

  Wonder why you through in the "in general" caveat in there? Is it because HTTPS 4 way handshake (2 full roundtrips) is actually an enormous hit on performance on high latency networks. And what are these high latency networks? Mobile networks of course.

I keep proposing "<a href="..." hash="sha256:abcdefg..."> as the quick, easy 80% solution to this. Works across sites and lets you authenticate resources you don't own.

This is the first I've heard of this. Is this currently in the HTML specification or in actual use, or are you proposing such functionality be added to the spec?

It's called Subresource Integrity. I just implemented it on my blog yesterday. It only works on CSS and JS, but image support is supposedly coming. I highly doubt that it would be a good idea for hyperlinks.

https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

I would argue it's a bad idea for images already. Images give away a lot about the type of content that you're viewing, which is possibly sensitive information.

Consider, for example, a Chinese computer connecting to Wikipedia over HTTPS, which then downloads an image over HTTP without S that is clearly related to the Tiananmen massacre.

I'm not saying that it's a replacement for HTTPS. Nothing stops you from using SRI hashes for HTTPS content. If you're concerned over censorship, you should be using HTTPS, period. SRI is more concerned about bad CDN content. Your CDN can use HTTPS, but your server's page says what the hash should be, so if your CDN goes rouge, things will start breaking.

Worth noting: subresource integrity doesn't currently change the mixed-content algorithm at all. An HTTPS site can't load an HTTP + SRI script.

It's useful for increasing the status quo on HTTP pages, or for defense-in-depth on HTTPS pages, but it doesn't actually enable you to do anything you couldn't before.

The hard part is delivering the initial page with those hashes. If you can deliver it over HTTPS, why can't you deliver everything over HTTPS? (There are good answers to that question, to be fair.) And if you can't, what use are the hashes?

> If you can deliver it over HTTPS, why can't you deliver everything over HTTPS?

> Works across sites and lets you authenticate resources you don't own.

These days, the significant performance hit of HTTPS is in the handshake and round trip times, not the symmetric encryption. Any modern CPU does AES much faster than the internet connection most people have. With CPU/hardware support, the cost is literally negligible. TLS_RSA_WITH_NULL_SHA likely would not provide any significant performance benefit, due to the handshake.

The only user agent SSL Labs reports as supporting null ciphers is Safari 6/iOS6. https://www.ssllabs.com/ssltest/viewClient.html?name=Safari&...

"Any modern CPU does AES much faster than the internet connection most people have"

But the server side, which is serving thousands or tens of thousands of these 10Mb/s connections, is a bit more challenging.

For high-bandwidth CDN applications (like Netflix, Youtube, etc), the drawback to SSL encryption is indeed due to the bulk data encryption. The penalty is twofold. First, the need to encrypt the data requires that the CPU have access to it. Even discounting the costs of computation, the need to touch the data removes a lot of benefits of sendfile / splice / etc. There is an interesting Netflix paper about this: https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf

Due to the nature of SSL, applying hardware offload solutions is problematic. The most efficient hardware offload would be a PCIe card that does encryption AND TCP offload. But then you're dealing with TCP offload, which is seldom a good idea. If you use an encyption-only hardware solution like the Intel QuickAssist cards, then you wind up in danger of running out of PCIe lanes. Eg, to serve at 100Gb/s per CPU socket, you need 16 PCIe Gen3 lanes for network, 16 PCIe Gen3 lanes for storage, and 16 PCIe Gen3 lanes for Quick Assist. So you're up to 48 lanes, but a Xeon e5-xxx v3 CPU has a max of 40 lanes.

They probably need to automate the "upgrade to latest" box, current version would be OpenSSL 1.0.2f 28 Jan 2016.

Anyway, I am not convinced the `openssl speed` benchmark output conveys anything to the people addressed by the site.

You could always fix it and submit a pull request. Then automate that.

https://github.com/igrigorik/istlsfastyet.com/pulls

Thanks for the hint, I did not realize it was on GitHub. Pull request sent.

https://github.com/igrigorik/istlsfastyet.com/pull/97

Love stuff like this. There's so much momentum around TLS lately, so great to see.

TLS is still slow, for users on 3G phones, remote areas, and 3rd world countries. But who cares about them? They're poor! They can cope with a slow web.

If I go to a site called IsTLSFastYet.com, I expect a huge bit of text that says "yes" or "no". Otherwise why use that format?

thanks for sharing. useful.

Reply on news.ycombinator.com