Comment by lucb1e | Hacker News Reader

Comment by lucb1e | Hacker News Reader

Parent Full thread

For those who want to understand how it works (it took me a minute, so I'll try to explain it simpler):

In simplified terms, the server usually stores a public and private key, and sends the public key to the client. The client generates a random password, encrypts it with the server's public key, and sends it to the server. Only anyone with the private key can decrypt the message, and that should only be the server.

Now you don't want to hand over this private key to Cloudflare if you don't need to, because then they can read all traffic. Up until now, you needed to.

What they did was take the private key and move it to a keyserver, owned by your bank or whomever. Every time the Cloudflare server receives a random password (which is encrypted with the public key) it just asks the keyserver "what does this encrypted message say?" After that it has the password to the connection and can read what the client (the browser) is sending, and write data back over the same encrypted connection. Without ever knowing what the private key was.

The connection from Cloudflare to your bank's webserver and keyserver can be encrypted in whatever way. It could be a fixed key for AES, it could be another long-lasting TLS connection (the overhead is mostly in the connection setup)... this isn't the interesting part and can be solved in a hundred fine ways.

Edit: Removed my opinion from this post. Any downvotes for my opinion would also push the explanation down (which I hope is useful to some). I mostly agree with the other comments anyway.

Thanks for explaining this. I would also love to read your opinion. Would you be willing to share it in a separate comment?

(And hopefully it doesn't incur any downvotes and readers understand it is just an opinion.)

Sure. TL;DR: I think it's mostly useless. From an engineering perspective this is a creative way to circumvent management's requirements ("our encryption keys must be kept on premises"), but otherwise I don't see the value.

Let's first see what the problem is with the old method, storing the private key on the Cloudflare server:

1. Cloudflare can read all traffic.

2. Cloudflare can read traffic they can intercept, even if it's not directed towards their server. Think public WiFi access points.

3. Anyone able to break into Cloudflare, physically or technically (or legally, hello NSA!), has the same capabilities.

Number two is a bit far-fetched without taking number 3 into account, except that there might be national security letters to obtain the private key, after which government agencies can use their own intercepts. Still slightly far-fetched, but a plausible concern if you're the bank that the next Edward Snowden just happened to use (Lavabit's private key needed to be handed over, disregarding all other customers' privacy).

Now we implement this new system, where you can only obtain the decryption for the encrypted session key (as sent by the client, e.g. a browser).

1. Cloudflare can still read all traffic.

2. They can no longer passively read any traffic, or hand over private keys to law enforcement. However, the encrypted session keys could still be queried at the keyserver. The keyserver would need to actively maintain a blacklist of session keys that are not in use at Cloudflare, e.g. by having webservers send them as they were used, and raise alarm flags if a false one comes up. But I'd need to work out this attack scenario further to be sure there is no loophole that could make it look like Cloudflare traffic even if it's direct (but intercepted by whomever) traffic. I think something like that might be possible, but I'd need to work it out to be sure.

3. Anyone hacking Cloudflare is now limited in the same way Cloudflare is (so the same as point number two).

So far the security concerns. Now as for speed and DDoS mitigation...

- Cloudflare now needs to query an external server (if it's in their datacenter, it can be pretty much considered under their control) for every https connection. Assuming there is a direct peering connection to a datacenter next-door, this would only be a couple of milliseconds in network latency, but so is DNS and building a TCP connection and fetching a page... it all adds up and in the end you notice that pages need to load, even if only for a sec.

- The SSL termination point is now nearer to the client, as Cloudflare has datacenters all around the world. You could in theory build the keyservers next to many of them, but then you're rolling a CDN for a CDN (a delivery network of content (keys) for a content delivery network (cloudflare)). Perhaps not a bad idea, but it just sounds a bit silly. All things considered, at best the connection gets only a tiny bit slower.

- Cloudflare is still useful for mitigating attacks, just like with normal http traffic. I mean, they can read the data, so it's more or less the same as with unencrypted http. The difference is the keyserver: they might run over capacity. Luckily you can now solve that problem by throwing money at it (you can deploy more keyservers, Cloudflare already has enough front-end servers to handle the requests). So it's now a money issue, not a technical one, which is how you want things to be.

- Session reuse makes this somewhat less of a pain, as the keyserver need not be queried for repeated https connections. We'd need to see numbers to know how much of a win this is, though. In any case, it's never faster than not having the keyserver.

So in this area it's still a win, but so was handing over your keys. Having keyservers makes it slightly less of a win.

---

In conclusion, you have three options:

- Don't use Cloudflare. Fastest website, but risk of DDoS.

- Old-style Cloudflare https setup: handing over your keys. DDoS risk mitigated.

- New technique: providing keyservers. Slightly slower than the old style, but with a tiny security advantage that might help with either management or rare kinds of high-profile attacks.

Disclaimer: I'm not a Cloudflare tech, I'm a software engineering and information security student (and a major nerd in general, I might add). I might be missing something, but many other comments seem to be saying similar things.

There's also:

- If you discontinue using CloudFlare, you don't need to generate a new key-pair, and all of the issues associated with the infrastructure for revoking the current key-pair.

- There is also this: https://news.ycombinator.com/item?id=8336041 , which mentions protection against key-recovery due to Heartbleed-type bugs, which is true. If the SSL-stack is attacked, the attacker only has the same access that CloudFlare does. They can't recover the keys from the key server.

One more thought occurred to me (I didn't see it mentioned around here?) that you keep some control of the relation with Cloudflare, you can pull the plug instantly at any moment you wish if need be, and because of that, they in theory must be more "on their toes" all the time, can't become spoiled too easily. Or at least should they become, you have some concrete leverage. Also if some competition for CF arrives, you now have the freedom to switch.

> From an engineering perspective this is a creative way to circumvent management's requirements ("our encryption keys must be kept on premises"), but otherwise I don't see the value.

This is a little bit like saying "From an engineering perspective, gold is a good conductor but otherwise I don't see the value."

Whether the desire to avoid sharing SSL keys with 3rd parties stems from management, regulatory or contractual requirements is irrelevant. The bottom line, from CloudFlare's perspective, is that they have been unable to address a significant and potentially highly-profitable market.

Keyless SSL might not be groundbreaking from a technical/engineering perspective but, as a service offering, it could unlock hundreds of millions in revenue for CloudFlare.

> Session reuse makes this somewhat less of a pain, as the keyserver need not be queried for repeated https connections.

If a DOS is actively attacking the bandwidth of the keyserver, none of the attacking connections will reuse old sessions, though.

True, but I meant for the performance for legitimate users.

Is this not subject to side channel attacks (for recovering the key)? Also, whats the point if you are letting a third party read all of your traffic at any rate?

For starters, you can move away from CloudFlare, and you don't need to worry about issuing a new key-pair (b/c CloudFlare, or a unscrupulous CloudFlare employee could use the old private key to impersonate your site).

Now you don't want to hand over this private key to Cloudflare if you don't need to, because then they can read all traffic.

Generally the key you would give them is for, and limited to, the resources that they cache/reverse proxy, so the same "read all traffic" concern exists.

What Cloudflare did is essentially, as others have mentioned, PKCS11 over the internet. PKCS11 is an existing, very well proven technique of sequestering the key in a hardware device, that hardware device doing what Cloudflare is moving to the client location here. It's neat enough, but they seem to kind of exaggerating the innovation a bit.

I think the exaggeration is because its a sales pitch, but it illustrates an important concept -- doing something "known" but in a generic way can be more valuable than having a proprietary solution. Of course someone could come up and offer a 'Cloudflare' box that just does key sequestration and sell that.

And while I think banks and others will find this useful, its even more useful if I can do it from a far away place. When you create an SSL connection in Europe to a west coast server there are a number of round trips. If those can be avoided you can cut the latency of the transaction. That is a good thing too.

> When you create an SSL connection in Europe to a west coast server there are a number of round trips. If those can be avoided you can cut the latency of the transaction. That is a good thing too.

Note that this will at most cut the connection setup time by half, as your premaster secret (RSA) or handshake data (ephemeral DH) still has to cross the Atlantic and get back to you (and the TLS handshake has 2 roundtrips).

Furthermore, because of

> Generally the key you would give them is for, and limited to, the resources that they cache/reverse proxy,

all dynamic data still is supposed to come from your server. So the browser will have to wait for that, while everything from the European mirror is cached anyway.

It is very obvious... In hindsight?

Not sure why all the negative reactions here... Was anyone else doing or providing this type of "Keyless SSL" setup?

You'd think if this was a known technique, the mentioned banks would already have been asking for it, implementing it, or doing it.

Personally, I think CloudFlare is one of the few companies on here doing innovating stuff, and solving real issues.

And if not - if they've pulled the wool over my eyes - then at least I can respect their marketing.

PKCS11 and Hardware Security Modules (HSM) have been around for a long time. There's in wide use in companies like NetFlix and Square including "over the network". This is not new.

You'd think if this was a known technique

Isolating the key for security/control purposes is a well known technique. It's what ssh-agent/putty paegent does, where you can even forward key usage over the network to different machines. It's what PKCS11 does to SSL/TLS, where generally the key is a hardware security module (CAs and banks don't have keys sitting on their public facing machines, but instead generally have dedicated hardware for it -- they are generally doing, at a different scale, what is described), though there have been various implementations and proposals over the years to do it over the network (e.g. https://svn.opendnssec.org/docs/p11proxy.png and pkcs11-proxy).

Why isn't it commonly done as Rackspace proposes? Well it's kind of an odd need for limited gain, but is effectively PKCS11 used in a different context. If they are solving a specific need for some of their clients then fantastic, but it's the narrative of the innovation that makes it a bit weird, and yielded a lot of the purportedly negative commentary.

Sometimes the innovation is merely being able to rephrase the apparently impossible question into the one that has an obvious answer.

It's worth remembering, no-one else was offering this or doing this... no-one else had phrased the problem such that the answer (even though it was known) became apparent.

> Generally the key you would give them is for, and limited to, the resources that they cache/reverse proxy

That got me wondering: Why don't attackers then only DOS the dynamic pages? Those that can't be migrated easily, like the login?

The dynamic pages still go through CloudFlare; there's a Web Application Firewall which can block abuse directed at a specific URL, pattern, etc. The cache is only one part of the DDoS mitigation.

(We've been working on a lot of great updates to the WAF for just this kind of thing)

Generally with CloudFlare people put as much of their site behind the CloudFlare proxy as possible, for origin-hiding and other reasons, even if we can't cache it (yet).

Reply on news.ycombinator.com