CloudFlare, SSL and unhealthy security absolutism
troyhunt.com
troyhunt.com
In particular, the "full" TLS they offer that Hunt is taking advantage of (because he can't deploy a working certificate) is grievously insecure: he's essentially running his system on top of the equivalent of the "goto fail" vulnerability in Safari that the Internet was (justifiably) freaking out about two years ago.
Hunt is smart about a whole lot of things, but he's wrong about this. There's nothing "unhealthy" about the absolutism here: you either have end-to-end cryptographic security, or you're vulnerable.
Or you can bring your own certificate from someone like Let's Encrypt.
If CloudFlare wants to live dangerously with origin connections, fine... but give end users a way to drop the connection if it isn't secure, like our browsers would normally.
I'm not sure why he talks about absolutism, where this usage basically invalidates the whole point of TLS. This seems like an after-rationalization on why he chose to use CloudFlare, there is much more reasons not to use them. In the end of course it depend on what type of site you use it for.
1) using the real hostname of the website
2) using the hostname of the backend website - assuming you supply a url with a hostname to cloudflare. however, this hostname in the backend url is used in the Host header which ends up causing lots of problems.
3) using the hostname of the backend website - but the hostname of the real site is used in the Host header.
4) explicitly let you choose the servername you want to verify
imo. it would be nice if cloudflare gave you the option between 1) and 3) or 4).
EDIT: oh. just noticed the cloudflare reply with the link to: https://blog.cloudflare.com/cloudflare-ca-encryption-origin/
I.e. you generate your keypair and produce a self-signed certificate, you should be able to give your public key and the actual key would be used for authentication. The subject (and SANs) of the certificate are ignored so it could be like "/CN=localhost" - only the key material is what matters here. That would be secure - in a sense third-party attacker won't be able to sniff what's between you and CloudFlare.
(Or you could generate your own (self-signed) CA, generate a certificate signed by this CA of yours, and give CloudFlare your CA's public key. This makes sense if you have multiple hosts and you don't want to share the same key.)
The only way to make this "less bad" would be if there was a way to determine if the backend infrastructure was in fact loaded end-to-end via TLS. Perhaps some non-cachable content? Not sure what the right answer is here.
It's easy to detect when Cloudflare is being used. There are multiple ways.
They're positively surfer hostile.
I don't have even the smallest idea how to configure one or what software I actually need, BUT, my greatest fear is that I'll end up with a pretty "broken internet" since I won't be using a residential IP anymore.
Cloudflare will just think that I broke into some random wordpress site and started using the server to ddos (who? with 1-2 requests/minute?).
It doesn't look that they implement any kind of serious IP reputation algorithm. I have a static residential address since I remember signing the contract, almost a decade ago, pretty paranoid about security with a clean PC and I never seem to stop getting captchas.
Is PIA a paid only service? I doubt any "hackers" will be using it to break cloudflare sites. On the contrary, I'd argue its users are actually more civilized than the residential folks.
I've no idea how clever CF are (tempted to quip clearly not clever enough) to decide their sites or their network is under attack. I'd imagine there's going to be a lot of connections from PIA's users, from all the various regional exits. Same for all the other VPN companies. I can't believe they can't be distinguished from a real DDOS or whatever else they see as "bad traffic".
Where Cloudflare start winding me up is they treat each domain alone. I get a captcha on blog a, 30 seconds later I'll get another at company b, 5s another at blog c. If CF gave me a 3 or 24 hour cookie I'd hate them so much less! (There's a couple of ways around captcha madness - If I switch to another exit I can usually get a captcha free one)
PIA is paid, but good value - around $35 a year.
Stepping on to soapbox... TLS needs to be replaced with something simpler, with fewer "buttons, dials and knobs". TLS pontificators need to separate authentication from encryption. They are two different problems, and it's possible we may only have a satisfactory solution for one of them. If that's the case, and it's the former problem that remains unsolved, then it could be worthwhile to reconsider the relative value of encrypted "channels" versus per packet encryption.
Hunt seems to posit that the above ratio is small, I take it you disagree?
On a side note, I don't get why cloudflare doesn't let you pin a public key through their interface; that would remove the need for CA signing. Considering you have exactly one client (cloudflare) connecting to your server, it's not like key distribution is hard. Of course this is moot now that cloudflare offers you signed certificates for free, but would have been a simple solution.
It sounds like Hunt expects to not see much interception and manipulation at the "backbone." My own experience suggests that such is common at national borders, and becoming common at more and more ISP borders.
like different shades of beige, but in the end it's still beige ... [0][1][2][3]:
[0] https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflar...
[1] https://blog.torproject.org/blog/trouble-cloudflare
[2] https://people.torproject.org/~lunar/20160331-CloudFlare_Fac...
[3] http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-hav...
Specifically they:
- Inject javascript into your code while returning pages.
- It modifies the code of your pages.
- Modify the http headers of your pages.
Cloudflare is a pest. They hired smart people to do stupid things. What's worrying is that a smart guy like Troy (who I hugely respect and follow on twitter) doesn't see through their BS.
FUD.
Not FUD. It kind of sucks if $POTENTIAL_BIG_CORPORATE_CUSTOMER can't access your site because it's blocked as a porn site, just like it sucks if they don't get your emails because your entire ISP got blacklisted by spamhaus.
CF issues tend to be browser-related, tor-related or configuration-related. I suspect these three happen far more than the odd organization that blindly blacklist websites at the IP level... {{cn}}
Have you ever used CF? They've constantly got local downtimes, that's not caused by any of those.
You mention "tor-related" issues, but Tor users don't even get most of the captchas (or even visit very many websites). Most of the captchas are shown to normal users on "bad" ISPs (i.e. not big enough to warrant whitelisting, so smaller local ISPs and business ip ranges).
And blacklisting sites on the IP level is nothing out of the ordinary in your typical BigCo.
Does anyone know what Cloudflare's response was regarding them treating Tor traffic suspiciously by putting up never ending captchas?
Cloudflare does a really shitty job at that though, I've literally never seen them beat a reasonably configured nginx instance running on GCE or even OVH. (That is for a single instance serving both EU and US markets).
It's honestly baffling that anyone who isn't drowning in bandwidth bills would use them, but I guess antiviruses and PC optimizers are a big industry too.
using subdomains one can easily partition bulk traffic which needs caching from secure traffic that requires end to end encryption, and benefit from having a tenth of server hits
sure everyone can host his caches and be better off. but that cost money, and localizing traffic to reduce latency costs even more money. hard to beat free.
My entire comment was about refuting this.
You don't need to host your own caches to be better off, you'll be better off by simply not having cloudflare in front of your server (even for transatlantic pageloads!).
Edit: I'll also admit I love CF as a customer. It makes things fast and easy. But it's concerning. Sorta like every time I use Google search.
Jet.com had a similar issue. I'm pretty sure CF customers are not getting sufficient info/onboarding.
Have you considered that presidential candidates may be subject to more malicious traffic than a typical site (and thus may considering adjusting their settings)?
> I'm pretty sure CF customers are not getting sufficient info/onboarding.
I can ensure you that Enterprise customers are assigned highly technical resources during onboarding that walk them through settings. Is there a specific suggestionyou'd like me to pass along?
Jet.com gad the same issue - captcha every time I erased cookies, despite having my own residential IP. Till I brought it to their attention. (Maybe coincidence.)
This mirrors my own experience as a low end user. Devs in Easter Europe would get challenged a lot until we went and whitelisted everyone.
It's a matter of customers not bothering. You can only hold a customer's hand so closely.
I highly doubt that CF customers understand. I cannot imagine them saying "yeah, I think requiring people to solve a puzzle before they read a text document seems reasonable".
As an anecdote, I get ~1 captcha per month because I live in Eastern Europe. Add to that Linux, Firefox, NoScript and you're in for a fun ride.
I absolutely hate this, you go to a completely benign website but you can't see the content unless you allow the injected CF to run in your browser.
It is of course theoretically possible for any site operator to pass information that was submitted securely, insecurely over the open internet. It probably happens, and we should look for ways to eliminate that possibility. But the fact that we can't eliminate all the possible ways that might happen is not a good reason for CloudFlare to make it very easy for site operators to do.
"Flexible" does prevent a small class of attacks that are possible against plain HTTP. But it does so at the cost of completely invalidating HTTPS URLs, padlock icons and so forth. Making it easier for small site operators to get a bit of security against the most simplistic attacks is not worth making it impossible for users who are serious about security to have good security. Worse, the benefits accrue to CloudFlare and their customers, but the costs are shared by all internet users. It is an antisocial practice and it needs to stop.
Or as Troy phrases it: "protect against the 95% (or thereabouts) of transport layer threats that exist between your visitors and your origin"
Would you consider your use here of the phrasing 'small class of attacks' justifiable? If so then you should describe your threat model in a bit more detail.
1) Amateurs 2) Professionals not targeting specific individuals (i.e. spammers, credit card fraudsters etc.) 3) Professionals targeting specific individuals (i.e. corporate espionage, government-like threats)
Let's go with the article and assume that category 3) is virtually impossible to defend against, and ignore them hereafter.
People in category 2 are concerned about volume efficiency - they will perform those attacks that are cheapest per CC#. They don't care about handcrafted attacks against individuals, or even edge networks of a few hundred people. Their main attack on unencrypted HTTP will be passive listening on big backbone connections, which is achievable with their resources, and which "flexible" does not protect against - indeed it makes people much more vulnerable to.
Who are the people using Firesheep and similar tools? Technically-inclined students playing pranks (maybe a decent chunk of "threats" by number, but not really a threat you need to defend against)? I just don't see it being used as part of serious attacks. No carder is going to bother. No spam-information-harvester is going to bother. Maybe people with personal grudges who are at just the right level of technical skill? But how common is that? (This can vary a lot - I can see that it would be an issue for public figures with controversial political positions, victims of stalking etc. - but not for the typical "person in the street")
In other words I think category 2 is the vast majority of (damage-weighted) threats; the threats Flexible protects against are mostly irrelevant, and the threats that Flexible makes worse are precisely the most important class of threats for most "ordinary people".
Who are the people using Firesheep and similar tools?
And that's from commercial ISPs - if free public wifi or tor exit nodes MITM traffic, that's not even newsworthy :)
Whether you think cloudflare and its backbone suppliers are above such activities is a matter of opinion, of course.
[1] https://www.eff.org/deeplinks/2014/11/verizon-x-uidh http://www.icir.org/vern/papers/header-enrichment-hotmiddle1... [2] http://arstechnica.com/tech-policy/2013/04/how-a-banner-ad-f... http://www.infoworld.com/article/2925839/net-neutrality/code... http://www.theregister.co.uk/2014/09/10/comcast_using_javasc... [3] http://thenextweb.com/insider/2016/08/03/comcast-isps-should... [4] http://news.bbc.co.uk/1/hi/uk/7770456.stm
That sounds like a good reason to discourage website owners from allowing ISPs like Cloudflare to run massive MITM operations on billions of "secure" connections all at once.
It's mostly possible for a skilled team to build sites that can defend against targeted professional attacks (e.g. crime gangs) - at the very least you can make your systems expensive enough to attack to dissuade them, and (depending slightly on what you're doing), potentially completely inaccessible to them.
It's not easy to do that - you need to design your systems with many layers of defence, e.g. you might design your systems so that that 3 or more independent and unrelated 0days are required in combination, which is likely out of reach/budget for most targeted attacks.
However you also included nation states in the category. Defending your systems against targeted attack from a sophisticated nation state is so close to impossible for most groups of people that it shouldn't be in anyone's security model[1].
While I don't agree with many points in the article, there IS a concept of unhealthy security absolutionism (nothing to do with cloudflare, though). Parts of the infosec industry have a tendency to declare things broken and unusable if we suspect possible compromise by a nation state - we should not do this[2]. VERY few users can reasonably expect to defend against these attackers, and telling people not to use $tech because it's suspected to be vulnerable to this is a disservice to users.
Note that the above strictly relates to TARGETED attacks by nation states, bulk collection and surveillance is very different, and more practical to defend against.
1: For groups that DO need to defend against nation states, this doesn't apply - but then for those groups there is no substitute to having highly skilled people directly available. A bit of googling isn't going to save them, and thus simply having a footnote of "If you're defending against the NSA, this doesn't apply" is sufficient.
2: An example is the NIST ECC curves such as P256 - around the time of the Snowden leaks, there was some speculation that the constants used for these curves were chosen by the NSA to make them subject to some unknown attack, and therefore some (even some respected sources) advised users not to use them. This, in my opinion, was bad advice for most people.
With regard to the NIST curves I understood there was no advantage to using those curves - they were slower than alternatives whose constants did not carry the threat of NSA interference. I agree with the general point that there is some absolutism going around (e.g. I think treating self-signed certificates as worse than plain HTTP is wrong).
The padlock doesn't mean anything at all and never has. It's snake oil from an era when e-commerce wasn't trusted by the general public. It only ever existed to encourage/trick technologically illiterate people into handing over their credit card details to websites when the pervasive public opinion was that entering your CC number into a website was dangerous. It was very successful in achieving that aim.
In 2016 it has no purpose and browsers should remove it.
I can have a red line through mine and be just as secure as you with a green padlock.
This is what the ap is speaking of when he mentions snake oil. You had to pay money to have people believe you are safe.
Now of course you don't, but what does that really get you?
I'm not going to bag on the people behind let's encrypt (they've done an amazing job getting a lot of people on board to a worthy cause), but really the only reason they exist is because browser developers decided to add a green lock in the first place.
I don't agree with that. Security is a matter of cost to the attacker. A green padlock adds a significant cost compared to a red line.
On the other hand, opportunistic encryption as described can also be misleading and dangerous. Yes, most attackers will have a harder time achieving MITM between CloudFlare and your EC2 hosts than an unsecured Starbucks Wifi, but the possibility remains. And in the case of the non-"(strict)" versions, there is no way for the user to tell that the data was not encrypted. The security community has spent a lot of (well-placed) effort training users to look for the padlock symbol before trusting a website. CF's opportunistic encryption undermindes that. Files can be served over insecure HTTP to CloudFlare, and even a competent end-user will have no way to tell.
As Troy mentioned, Let's Encrypt and CF's Origin CA are steps in the right direction. Trust On First Use could be another one, allowing for a partial departure from the CA model. In any case: opportunistic encryption is a big improvement over no encryption at all -- but it should be clearly recognizable as such, and must not be confused with identity verification.
Agree. If the CDN is compromised, all bets are off. In my opinion, CloudFlare Flex/Full are a lot more vulnerable to attack (because the resources can be obtained insecurely over the public internet) than others (where TLS is either terminated at the network boundary and encryption is only lacking thereafter, or where resources are pushed through secure connections over the Internet).
> as opposed to just being secure on your network.
I'm not sure it is a reasonable thing to ask end-users to understand how many networks are involved in serving their content. I also don't think the situation would benefit from CloudFlare (or any other proxy) announcing which parts of the connection they encrypt. Instead, I think CloudFlare should be more strict in their origin connectivity: only accept encrypted data, and if the origin certificate is not a CA-issued one, Trust on First Use and ask the site admin to verify updated certs. CF is also in an excellent position to adopt a perspectives [1]-like approach, if an MITM is suspected.
Overloading it to mean something less than client to source is disingenuous and how we ended up with the infamous NSA Google unencrypted back-haul slide.
10 years ago one might have been able to make a "good enough" attempt argument, but nowadays I don't see how one can't include a nation-state backbone adversary in ones encryption calculus.
Any step away from the horrible "HTTP is fine, but god help you if using a custom certificate" scaremongering currently implemented by browsers is good to me.
I absolutely cannot understand this: if is's a trusted, non expired certificate, show it as green. If not, there's nothing more insecure about it than using plain HTTP; should we display huge scary warnings for plain HTTP, too?
https://www.chromium.org/Home/chromium-security/marking-http...
I agree with the parent post: both cases should trigger a non-intrusive indication of non-security, e.g. a broken padlock. Scary warnings are in order if a site was previously using a trusted cert and has either stopped doing so, or changed its (untrusted) public key.
Trusted certs are a sensible way for a trusted level above that. They should be supplemented with HPKP as well, and scary warnings if the secure cert disappears.
I'm arguing for the browser to have the same behavior for all insecure connections, be they HTTP or HTTPS on a self-signed cert. There clearly shouldn't be an indication of "security" (these should continue to be reserved for the CA-issued and particularly EV certs), but I agree with GP in that there shouldn't be a scary warning message either.
The scary warnings have their place: if a page that was previously secure (i.e. CA-signed cert) stops being so (either because it is now being served over HTTP [potentially sslstripped], or because it now uses a self-signed cert), then a warning should be issued.
Finally, we already have HSTS preload lists to bridge the gap before first access. If a site is on one of these and served insecurely, a scary warning is appropriate as well.
That's why self-signed certificates generate warnings even though HTTP connections don't.
Just get a real certificate, or have your users add your certificate to the trust store manually.
Right now, attackers can quitely turn off SSL for most sites if they just strip it from the connection; requesting over HTTPS and re-serving over HTTP. The majority of users types "example.org" into their address bar, not "https://example.com". Most users won't notice, especially not if the favicon is replaced with a fake green padlock.
> Just get a real certificate, or have your users add your certificate to the trust store manually.
Yep. Given the availability of Let's Encrypt (and others), the discussion is mostly moot anways :)
I always configure my Nginx instances to only respond to Cloudflare's IP ranges.
I tell Cloudflare to sign all communications with my origin server and I configure Nginx to validate those signatures.
I encrypt the communications between my origin and their system.
How can this be man-in-the-middle attacked even by a state actor? I'm ensuring that not only is the data encrypted between Cloudflare and my origin server, but that I'm verifying that nobody is able to impersonate Cloudflare.
I think we get a lot of criticism because we explain clearly what we are doing and people pick over it. There are always going to be criticisms but I'd rather we be transparent and take some heat than say little and clients not fully understanding how we operate.
Sure. With the correct settings it's fine. The problem is their "Flexible SSL" setting, which should be labelled something like "My site is not secure, but I want to lie to my users and pretend that it is." Yes, clients still have to choose the bad option, but CloudFlare should not be offering it to them.
This occurs everywhere on most every network and should not come as a surprise, although many technologists and researchers argue against it while others argue the principle is holding back advancements.
Am I missing an obvious reason for this not being an option? It must have crossed their minds at some point..?
(Edit: come to think of it, iirc cloud front does support wild card ssl for your own domain these days?)
I have the same use case with GitHub pages -- https://www.glowing-bear.org is Full SSL in Cloudflare, but I can't switch it to Full (strict) because the upstream certificate is for * .github.io
For example, if you are CNAME'ing www.example.com to example.anotherprovider.com, you /may/ be fine with us checking that the origin certificate has a SAN matching that destination, *.anotherprovider.com, or a hostname that you specify.
While Apple is mostly patched at this point, numerous flaws were recently discovered with how the CONNECT method was implemented, allowing an attacker to fully get into the middle of HTTPS communications in at least iOS and OS X:
Slightly less concerning, but still very bad, researchers also discovered methods for reading HTTPS URLs and cookies through WPAD vectors:
https://auth0.com/blog/heads-up-https-is-not-enough-when-usi...
Edit, typo
Edit: if there's an alternative to a Cloudflare-type MITM for fighting DDoS in layers two through five I'd love to hear about it
Well this deserves at least a comment -- I know this is a rhetorical comment: why on earth would a government need to be involved with copyright infringement to the level where now it's "exactly the sort of site". Are these websites used to plot mass murder / rape, treason and the like? Nope. Then why on earth would a nation state get involved with it?
I know also that malware is now abusing these cdns, so I have been forced to ban entire cdn ranges. I wonder if cloudflare has made any progress implimenting DNSCurve?
Feel sorry to read such a generic statement.
> Full (strict): CloudFlare issues the certificate and they'll intercept your traffic, but then it's all HTTPS to the origin and the cert is validated as well
Ah! Seems to say that it's ok if your certificate issuer were to be a MITM.
> What it means is that you can choose how much SSL you want ..
Interesting point. So privacy and data integrity between two parties can vary, the states being: (1) off, (2) flexible, (3) full, (4) full (strict). Amusingly, even when you choose option 4, i.e. full (strict), your certificate provider would act as a man-in-the-middle.
--
[It makes the entire article seem like a Cloudflare campaign. Thank you.]
We've made the generation as simple as possible—just two clicks—and also offer an API for users aiming to automate the process: https://api.cloudflare.com/#cloudflare-ca-create-certificate.
At some point during onboarding of new zones/hosts/etc. I'd like to start encouraging users without a (valid) origin certificate to install an Origin CA cert.
Curious to hear your thoughts on how else you'd "push it" more?
Oh, wow, so, you paid only 2 cents instead of 10 cents for bandwidth? What a huge advantage!
I mean, seriously? Where do you buy your bandwidth that it's a relevant cost factor below 10 TB per month?
This isn't true anymore. I would consistently get SSL invalid errors from them when trying this setup. You have to have a valid SSL cert for CF <-> Your server now it seems.