Keyless SSL: The Nitty Gritty Technical Details | Hacker News Reader

Keyless SSL: The Nitty Gritty Technical Details

blog.cloudflare.com

162 pointsjgrahamc68 comments

Interesting how they specifically refer to yesterday's discussion here on HN and over at Reddit, but don't go into the frequently mentioned complaint that despite not having access to the PK itself, Cloudflare can still intercept and modify all cleartext sent between the client and server, which for most intents and purposes means pretty much the same. Maybe not intentionally so, but it comes off as slightly misleading, because many people I spoke to who briefly skimmed the original announcement thought "keyless SSL" would mean this is no longer the case.

Let me ask again: why not offer a setup where Cloudflare only acts on the network layer instead of on the application layer and proxies the still-encrypted HTTPS packets to the destination server? This would mean a customer could not use Cloudflare's caching (CDN) features, but still enjoy their DDOS protection without them being able to see all my customer's private details. If I were the CISO at one of the world's largest banks, that is exactly what I would want.

Because DDoS protection isn't just something you do at the network layer. It's important to be able to apply filtering at the "application" level.

I understand that there are many different types of DDOS attacks, and that for some of them having access to the content on the application level makes your job of mitigating them a lot easier.

But even without that, would comparing the encrypted traffic patterns for an individual client to other client's patterns (or to that client's traffic to one of the other 2 million Cloudflare websites); and analyzing the encrypted traffic as in [1] not still give you a wealth of information to use? It might have made for a much more rewarding two years of development: even if such a setup would be slightly less effective at mitigating DDOS', it would be infinitely better in terms of privacy and trust.

[1] http://arxiv.org/abs/1403.0297

In some ways what you propose sounds even scarier, i.e. figure out the best way to look inside SSL encrypted data without having the key.

But in terms of privacy and trust, CloudFlare is already in the position of terminating SSL connections and establishing new ones to origin web servers. We take the protection of that data and the associated keys very seriously. There will be more announcements over the coming weeks and months about how we secure things.

I'm not quite following how having just enough insights about the encrypted data to perform DDOS mitigation would be scarier than having full read- and write access to the cleartext. Thanks for the responses though, and definitely looking forward to those blogs.

You are implying something fundamental: that the encrypted traffic could be adequately analysed for insight without the need for decryption.

Yet to do so would be to defeat SSL itself, or at least to declare it as insufficient to adequately protect secrets.

What CloudFlare is doing isn't defeating SSL or any kind of attack on it. They are merely working around some prior limitations on requiring access to an organisation's private key.

As a proxy that is charged with DDoS protection (and other types of protection and performance improvements), they are being asked to terminate and work on the unencrypted data to a very strict set of complience by the end organisations, but they need to do this in a way that does not involve possessing or having access to the private key.

Their solution works extremely well given the multiple constraints (technological and legal) that they have.

  > You are implying something fundamental: that the encrypted traffic could be adequately analysed for insight without the need for decryption.
  > Yet to do so would be to defeat SSL itself, or at least to declare it as insufficient to adequately protect secrets.

This is possible through HTTPS traffic analysis, see [0] and [1] for starters. Of course, it's much easier for Cloudflare to do analysis for DDoS protection if they have access to plaintext.

Whether this means that SSL is, as you say, "insufficient to adequately protect secrets" is an interesting discussion to have.

[0] http://arxiv.org/pdf/1403.0297.pdf

[1] http://blog.ioactive.com/2012/02/ssl-traffic-analysis-on-goo...

Actually thats pretty disingenuous. Most of the attacks CloudFlare prides itself on defending are amplification attacks (NTP and DNS specifically) not DoS attacks within the relevant protocols. Servers that simply forwarded 80 and 443 while dropping everything else sitting on really big pipes is really what people are paying you for today.

You are correct that we deal with amplification/reflection attacks all the time.

But what you don't see are the HTTP-level attacks where we put in place filtering rules in our WAF to block them. You don't see them because we mostly don't write about them. These attacks are different from the NTP/DNS style (which fill pipes) because they use server resources on the origin web servers.

We need to be able to defend against both.

Sure, there are probably a handful of custom rules that some customers get, but like I said... most of your customers are paying for something that drops NTP/DNS and forwards HTTP/HTTPS.

The standard WAF that all customers get scores pretty badly in independent tests: http://www.slideshare.net/zeroscience/cloudflare-vs-incapsul...

The WAF product has had a LOT of upgrades since that report, and has a huge update coming in Q4. That report is pretty out of date; I'd be happy to provide a test setup for Zeroscience to do a new analysis if they'd like, and am looking at how to do a continuous test/demo of the new WAF, because it's pretty interesting how it works.

When a source IP is shared (Tor exits, carrier NAT, etc.), trying to push as much into URL pattern vs source-IP filtering, for instance.

They can manipulate traffic, but so can the bank's operations division. The fact that Cloudflare is a different organization makes contractual agreements much more important, but as long as they are in proper order the difference shouldn't be of importance to the end user.

It's still based on implicit human trust though.

Well for that matter so is the assurance that only you can purchase a valid SSL certificate for a domain you own. At some point, you gotta trust some people!

The difference to the end user is that while the bank's operations division can only do this for a single website, Cloudflare can do so for 2 million of them.

That is indeed a difference, but perhaps not directly relevant to the bank's security.

For large scale traffic manipulation, Google is even better suited. They have arbitrary access to the DOM tree of over 10 million websites via Google Analytics, including several banks.

The problem that Keyless SSL is solving is the need for a client to hand over their SSL keys to Cloudflare in order to use Cloudflare's services.

It eliminates the risk that the key will be stolen from Cloudflare and used by the Bad Guys to set up a fake website to steal everyone's credentials.

You might not perceive how big the benefit of this is but you're not the target market. I used to handle the SSL encryption keys (and server security) for the online trading systems of a global investment bank. In my opinion, not having to hand over the SSL keys removes a significant obstacle to banks using Cloudflare's services.

Yes, Cloudflare will still be able to read and, in theory alter, the data flowing between the end-user and the client (I would not be surprised if a future enhancement to this mechanism enabled some kind of hashing/signing of the HTTP requests/responses, such that any attempt by Cloudflare to alter the content would be obvious) but that is an entirely different risk than the risk of your SSL keys being stolen.

CDNs are used to deliver static, highly cacheable content, like assets, images, media content, stuff that will be delivered to many users. It is very rare that a website serves this type of content under the same domain as the dynamic content (think cdn1.whatnot.com, assets-myapp.com). In fact for applications with a lot of assets this is a must because of domain sharding and to avoid the overhead of sending the cookies on every single request. Since you need a different domain, you can (you don't have to) use a different certificate, so it's not a strech to use a different private key.

Then there are the applications where security is more important than latency - online banking, payment gateways, tax filing. Those applications don't have too much cache-able content so they simply won't benefit from CDNs.

I also don't see how is this faster. Sure, CloudFlare Keyless SSL is faster than serving the content yourself, but sending that packet over the internet is always going to be slower than not. So now I have a slower (compared to the CDN having my key), more complicated option that requires me to run an additional keyserver and doesn't provide me additional security. The only upside of this seems to be good PR.

Depending how sensitive the content is, in fact if I could I would choose to deliver it over plain HTTP and sign it somehow. That would be best of both worlds.

Edit: I admit I totally missed the point about DDoS protection.

I think you may be missing that one of CloudFlare's key features is DDoS protection (and in fact it was a DDoS that initially caused the banks to approach them), including at the application layer. That functionality cannot be fulfilled without CloudFlare having access to the content of the communications, nor can any of their other application-layer security functions.

Cloudflare isn't just a CDN. They protect against DDoS attacks. That is what financial instututions are interested in.

You should read eastdakota's blog post from yesterday: https://news.ycombinator.com/item?id=8334933

This is something that I spent a lot of time on a few years ago.

Some of the HTTP accelerator servers I ran were in countries or sites that were more problematic than others, so it was desirable to not locate SSL keys on them. Instead of handling HTTP, the servers would become a TCP level proxy, and forward resp / req packets between a server with the SSL key and the client.

The biggest hurdle is that without being able to decrypt the incoming SSL, you can't know anything about the HTTP request -- including hostname or path. So you have to forward all of your received traffic to the same place, and you can only vary the destination based on the source IP that receives it.

IPs are limited. There is no way that Cloudflare could have even 1 IP for each of their customers in each of their sites around the world, it simply cannot be done.

There is lots of other common functionality that you lose (edge caching, cookie routing, etc.) but the inability to route at layer 7 is the big one.

To save people a long read, there's nothing particularly surprising in these details. The key is held by the origin; when a computation using the key is needed the origin is asked to perform it over a secure connection. They have improved support for session resumption in a distributed environment (which is even more important now that key computations are even slower), this is commonly done as closed-source, and CloudFlare have promised to open-source it.

Good improvements, and session resumption is important to implement, but nothing groundbreaking.

Oh well. Guess we'll have to wait for Homomorphic encryption:

http://en.wikipedia.org/wiki/Homomorphic_encryption

The Keyless SSL server is open source and available on GitHub - https://github.com/cloudflare/keyless - although I'm wondering if its dependence on OpenSSL was a good choice.

Out of curiosity what would you have used?

CyaSSL - http://www.yassl.com/yaSSL/Products-cyassl.html

GPLv2

My guess is that the customer for whom they developed this has crypto-acceleration hardware that is supported by OpenSSL. Also, this is only doing the actual crypto operations, not running the whole SSL protocol.

Edit: I guess it's also exposing SSL publicly for talking to CloudFlare, which is I guess why they suggested IP firewalling as well (in case of another HeartBleed).

And it's only a reference implementation!

You've hit on the most dangerous part of this whole scheme: increased attack surface. There is now a much higher likelihood of a successful attack taking down the site, due to the client needing to both open up and harden its keyserver on the internet (i'm presuming they don't have private connections between cloudflare and the customer site).

And obviously if any hole is found in either the client's keyserver or the network or host it runs on, the private key is now at risk. One could argue that the client is much less reliable in terms of hardening their security than CloudFlare is.

In order to reduce the attack surface, the client's keyserver needs to be connecting to CloudFlare, so they don't need to expose a service on the internet and thus an attacker won't know what to try to attack to take down the keyserver.

What? Increased attack surface? For the people that this matters, the alternative to this service is running your own https web server, where the server with the keys in its' memory is also running server apps that directly interface with the outside world, or at the very least a proxy through which all traffic is funneled.

This system reduces the attack surface to the minimum possible: a single oracle that takes encrypted data and returns it decrypted, all secured with pinned, internally signed client certificates where all connections are coming from a trusted partner and where the TLS security can't be downgraded.

To me, this seems like the maximum reduction in attack surface that is theoretically possible.

I'm talking about DDoS attack surface, not private key security attack surface. If you want DDoS protection, this scheme is less reliable than the alternative (having CloudFlare host the private keys).

To get the most secure private key security, a system like this would be the least attack surface only if the keyserver is connecting to the http server. By having the http server connect to the keyserver you expose the keyserver to attack, no matter what protections [like IP whitelisting] you put in place.

How would an attacker learn of the private key server? If that were easy, the very concept of CloudFlare would be undermined, which obviously is not the case. Having learned the address of the private key server, how would an attacker proceed? ISTM that all she could attempt would be to repeatedly try to set up the "secure tunnel" mentioned in TFA. At some point the private key server could just ignore those. I don't see much DOS potential there.

> How would an attacker learn of the private key server?

If they're using CloudFlare and they're a large financial institution we'll just assume they're using this new service. Now we just need to find the location of the keyserver.

There's a variety of ways to locate a server when you don't know where it is. The default is to scan the target's network. To find the network you can look at the ARIN info for addresses associated with various hosts under their domain and look at non-CloudFlare allocations. Or you could do a simple brute force search of DNS records (there's bound to be a DNS record for it so cloudflare can connect to it) and look for an interesting seeming host. Or you could look at the output of HTML for embedded hostnames in comments (pretty common for large sites). This is a very brief list of examples.

> Having learned the address of the private key server, how would an attacker proceed?

Once you have the keyserver address you can just DDoS it, and all of a sudden no new SSL handshakes can complete, meaning bye bye static content over SSL. The ISP'd have to null route all the DDoS traffic to the keyserver, which can take time. But that's just a network attack.

You can investigate further and attack the host/service. Maybe they forgot to whitelist the CloudFlare IPs and you have open access to the box via some random service, or the keyserver TLS port. Maybe you abuse BGP and find a way to spoof one of CloudFlare's IPs. Or maybe you social engineer somebody into adding a new address onto the stack of their IP whitelist. Or maybe it's "in the cloud" and you can attack the machine from the cloud provider's network.

Once you can connect, you can pick your poison with how you want to abuse or attack the machine, but there's almost no point when a plain-old DDoS will do.

All of these are the same attacks you'd use on any origin server for any CDN. The difference is that normally a CDN will keep serving cached static content if your origin goes down. With the keyserver down, you can't even serve cached static content, except for non-encrypted HTTP of course.

This is CloudFlare's entire core business model. CloudFlare is just a reverse proxy+cache. The original web server is still run by whatever company is using their services.

Fixed: "If they're using CloudFlare and they're a large [anything], we just need to find the location of the [origin webserver]."

This hasn't been a problem for any of their customers that do this so far (hint: all of them), adding a keyserver is no different.

It's different in that now attackers can take down even cached sites. Before it was only dynamic content that was vulnerable. It's the difference between having your home page up and serving cached content, or your entire HTTPS site being down.

To give you an idea how this affects people in the real world, some websites will make thousands and even millions of dollars an hour in advertising revenue and paid services. They depend on CDNs to handle the traffic of that many users and make it seem like everything's moving smoothly even in the event of a temporary outage. If that site goes dark completely, they lose tons of revenue, and people get fired.

In another case, let's say a large financial institution, they might need to provide authoritative and highly sensitive information around the clock to organizations that basically control the flow of money around the world. Downtime isn't really an option. Without Keyless, this information stays up, cached. With Keyless, an outage can make this information disappear, with potentially far-reaching global financial repercussions.

To reiterate: if you don't use Keyless, your (HTTPS) static content stays up under an outage. If you do use Keyless, your (HTTPS) static content goes dark under an outage. (For https clients that don't have an existing valid session ticket on the CloudFlare server)

You can always use plain HTTP and avoid the outage, of course. But for large financial institutions that's probably not an option.

Also, please note that i'm really not trying to be inflammatory. I'm just pointing out that this is a new, additional point of failure and it can have real consequences for the content people provide over HTTPS.

You make good points.

> It's the difference between having your home page up and serving cached content, or your entire HTTPS site being down.

To be fair, any clients with a valid session would see no difference between Keyless all the way down to plain HTTP (i.e. only static content). So the real difference between keyless and more typical setups is that new users can no longer see static content if the key server is down.

Given that the key server would see a vanishing fraction of the bandwidth and number of requests, in addition to its' extremely simplified and locked down API, I would guess that it's much more difficult to take down compared to a normal web server; you'd essentially have to take out the network equipment around it before it became overwhelmed itself. In addition, only a tiny fraction of legitimate packets would need to go through to be able to support a large number of clients. But perhaps I'm mistaken.

Now I'm curious how often this type of attack occurs, i.e. overwhelm the tunneled servers behind CloudFlare's back.

If the http server is compromised you're screwed either way.

If the SSL key is on the http server when it get compromised, you've just lost your SSL key. Very bad!

If it's not, the http server could DDOS the key server. Less bad! Just change the key server address and point a new http server at it.

No, the alternative in situations like these when you need the added security is to use an off the shelf HSM.

Open source but a rather restrictive license!

The source is published but it wouldn't count as a OSI open source licence.

Also note the patent mentioned in yesterday's discussion.

I don't get it. What's the point in keeping the key secret from cloudflare whilst providing a key server signing everything it is asked to sign? Isn't this like "Sorry I don't trust you enough to provide you a key to my apartment. But if you need something, just ask the janitor, he'll open the door any time you want"?

If you no longer trust the person, it's easier to tell the janitor not to admit him than to rely on his returning the key (and any copies) or to change the lock on the apartment.

Is "changing the locks" (revoking the certificate) really so complicated that this "janitor-solution" is easier/cheaper/safer?

The CA can revoke the certificate, but since revocation checking in browsers is neither universal nor reliable under attack, revocation isn't a completely effective way to recover from a compromised private key.

The more people who have a copy of a key, the more chances for a copy to be lost or stolen by a bad guy.

Why should all of these have a marketing spin to them? By that I mean a catchy name. Keyless SSL is very misleading. I guess after the stupendous success of the heartbleed bug marketing every one now wants a catchy name.

Having said that is it solving any security problems?

1. Companies don't have to give out their private key to Cloudflare but they still give them the encryption and hmac keys. So nothing really changed from a privacy or data protection point of view.

2. Now they opened up a new public API on the company's server for cloudflare to use the private key.

3. Is it simply to mitigate against another heartbleed kind of bug? Whats preventing the same bug to appear in the company's server?

Interesting read. Still no information how it compares to other HSM implementations however, and why they didn't use the more well studied PKCS protocols in theirs.

I would imagine the problem was more with existing implementations rather than some problems with HSMs per se, as they probably have much larger latency than you normally have so things like blocking reads start to matter.

I guess you mean PKCS#11, but that isn't a protocol. It's a C API. Trying to funnel it over a network would be both extremely complex, and non-standard.

No, I was thinking of PKCS#7 which you can run over IP if you want. But it's a complicated set of standards and I'm not sure how they inter-relate. That's why I summarized them as PKCS-style protocols.

This is not some theoretical thing. You can buy these devices off the shelf. If you have worked with PKI you have seen them, or some variant thereof.

Yup, I know. My day job used to be writing firmware for nCipher/Thales HSMs :)

Cool! We had those at a previous job. Good stuff.

Is this all the software necessary to run your own little keyless-like infrastructure with a couple of nginx server not having a key and talking to this reference key server over the Internet?

BTW I probably should have asked yesterday but which CloudFlare plans will include the keyless feature?

Sorry if I'm missing something, but shouldn't this be called "Keyless TLS" as CloudFlare uses TLS and not SSL for this program?

From the article:

"This may seem confusing at first, but makes sense since TLS is just a minor update to SSL 3.0. Subsequent versions of TLS have followed this pattern. Since TLS is an evolution of the SSL protocol, people still use the terms TLS and SSL somewhat interchangeably."

Oops, must have missed that. Thanks! I'm personally a fan of using the new official name whenever possible but I understand the tradition/ingrained nature of SSL.

So from a non-technical business owner perspective this means I can offer SSL for free without buying certs or making changes to my server?

No. This allows a content delivery network (such as cloudflare) to host content on your behalf without having to have direct access to your private key material.

The main use cases you may be interested in are to 1) make it so that users connecting to you get a faster experience - since they can connect to a geographically nearby cloudflare server, rather than your distant server and 2) make it so that cloudflare can absorb large denial-of-service attacks that your server couldn't otherwise cope with.

Ok thanks. So this basically makes it easier and more secure to get set up with cloudflare?

So why couldn't Cloudflare set up what I'm suggesting? Are there technical reasons, or just lack of demand?

Cloudflare probably is easier than running your own servers and configuring them correctly - I don't know since I haven't used them myself.

But it sounds like you're not very familiar with the importance of the private key. If anyone else other than the bank obtains the bank's private key, the bank would consider that a serious failure, since it means others could impersonate them. The whole point is that you shouldn't give your private key to anyone else, and that without that key, others can't impersonate you.

This "keyless SSL" scheme allows the bank to set up an entity it controls which knows the key. This entity delegates to cloudflare the ability to pretend to be the bank on a request-by-request basis, without divulging the key to anyone. If cloudflare gets compromised, the bank can stop that delegation on demand, the compromise is closed and the key is kept safe.

To serve SSL for your domain, Cloudflare needs a certificate for your domain issued by an accepted CA, which someone needs to buy.

The only way for them to set that up would be to become a CA, get browsers to accept their master certificate and then issue their own certs for their clients.

They actually do offer a service where they proxy your plain-text HTTP to end users as HTTPS and provide a certificate. It's extremely convenient.

Yes, but you still need to buy and give them a certificate, right? The question included "can offer SSL for free without buying certs", which is not possible, AFAIK.

No, they essentially add your domain to a list of domain names in their own certificate.

Really? And their CA allows that? That's amazing.

It's one thing to keep SHA1 in old services, but why do they keep pushing SHA1 into new ones?

From what I understand, the keys stay the same - this is the whole point of this approach. So it's not a "new service". I agree that SHA1 should be deprecated ASAP though.

What use of SHA-1 are you talking about?

From the user POV, nothing is changed. The decision to deprecate SHA1 or not isn't specific to Keyless SSL (ie, Keyless SSL isn't a new product for the users)

From the bank's point of view, where we can say that Keyless SSL is a new product, only 2 cipher suites are allowed:

    ECDHE-ECDSA-AES256-GCM-SHA384
    ECDHE-RSA-AES256-GCM-SHA384

which, as you can see, don't include sha1.

Reply on news.ycombinator.com