A public letter to CloudFlare to fix their snoopy vendor | Hacker News Reader

A public letter to CloudFlare to fix their snoopy vendor

github.com

417 pointscaptn3m0105 comments

Please Cloudflare, I'm a paying customer and have some IPv6 only users that are very frustrated every time they see a Cloudflare challenge page. Your provider, HCaptcha still do not support IPv6. I have to use workarounds like an alternative domain without CF and this is very frustrating.

It's a little hard for me to believe that there are IPv6-only users out there. Who are these people?

edit: I realize there are plenty users behind NAT64 gateways etc, that's the point: how many users out there have no IPv4 connectivity at all?

It’s super common for cellular networks. Apple even has a rule for the App Store to reject any applications that don’t work in an IPv6-only environment because of this. They’ve had this rule for six years.

Must be smaller providers doing this? In the US, we've got CGNAT IPv4 and regular IPv6 for cellular.

It's driven mostly by adoption outside of the US, particularly in Asia. In 2019 over half of the connections in India were IPv6 (of 475 million devices, 52.49 percent are IPv6 devices, according to the APNIC report).

That’s not the whole story. Your service doesn’t need to be accessible through ipv6. Just over 464XLAT. That won’t be a problem if you use high level system apis to make connections. All of my servers are ipv4-only and I haven’t had any apps rejected by apple.

You're partially right. Your apps need to support IPv6 because the internet connections can be IPv6 only. I believe T-Mobile is a big one does this. They do translation for their clients to the IPv4 internet, so they'll be hitting your IPv4 server through a translation layer. The client is still IPv6 only.

That doesn't make sense. If I hit amazon.com on my browser, which is IPv4 only, t-mobile needs to provide a (private) IPv4 IP to the phone to make that possible.

Edit: was totally wrong about this: https://www.internetsociety.org/resources/deploy360/2014/cas...

iOS (since IOS 12) and Android have native clients that can tunnel IPv4 requests over an IPv6 only network that are used for providers like T-Mobile.

Not necessarily.

T-Mobile could respond to DNS requests with an IPv6 address that includes an encoding of Amazon.com's IPv4 address in it so that when you try to connect it it, the gateway knows what IP address you're looking for and can do the NATing there.

Of course, this is just conjecture about a possible way of making it work, and it could easily get broken if you configure your phone to use a different DNS server.

Thanks, that's a good point. Apple talks about it here: https://developer.apple.com/library/archive/documentation/Ne...

People on mobile networks, especially outside of Europe and the US

Most users from KPN (largest dutch telco) are stuck with just a IPV6 address. Phoning home is often quite a bit of a challenge.

I think the parent poster isn't asking whether you only get an ipv6 address, but rather you only get an ipv6 address without some sort of CGNAT for ipv4

This is quite common in countries with lots of people and not enough IPs.

ipv6+nat64

Why don't they use IPv4 with CGNAT in addition to IPv6? That's what the US providers do.

NAT64 and NAT44 are both forms of CGNAT; NAT64 can be simpler to deploy.

You don't need two nats, you don't have to track two types of ip addresses per customer, for most of the traffic, you don't even need nat at all (youtube, facebook, etc, all support native ipv6), and statless nat64 can be simpler and less resource intensive than cgnat

> It's a little hard for me to believe that there are IPv6-only users out there

Why is this hard to believe?

IPv4 ran out a while ago depending on which part of the world you are in. New networks do no longer get IPv4 by default. Some can get very small allocations that are barely enough to operate nat64 gateways for a few thousand users. Quite a few networks decided to not invest in legacy IP any more and better spend their (limited) resources on other activities.

> Who are these people?

me + non 1st-world countries.

IPv6 without any transition mechanism is not Internet access. It's more like a tenth of the Internet. If an ISP has to buy a few addresses to run NAT64 that's a cost of doing business.

What happens when there are literally truly no IPv4 addresses left for sale? Not today’s reality, but the clock is ticking.

There will always be IPv4 for sale just as there is always land for sale.

This is the “oil will never run out” argument. Technically true, but irrelevant.

I chose my analogy carefully. Oil is consumed but land and IP addresses are not.

You can't really just "find" more IPv4 though. Cloud platforms are eating blocks for breakfast, lunch and dinner, and while it's always going to be for sale, there's no reason to expect you'll be able to afford it.

Fortunately, if you can't afford IPv4 then no one else can afford it either, which means the incentive to adopt IPv6 is extremely strong and thus you no longer need IPv4.

Existing large ISPs and cloud providers with remaining IPv4 addresses and no IPv6 support are not going to have a strong incentive until after the damage is already done.

Why won’t Google, Cloudflare, AWS, etc. simply own all most IPv4 addresses and leave all end users using NAT (or even CGNAT) to access them? This cements their businesses, since then nobody can reasonably host their own servers except through them.

It works both ways though, doesn't it? If you are IPv4 only, you can't access/can't be accessed from the whole internet. If you are IPv6 only, same applies.

No, because there is nothing interesting that's IPv6-only.

That depends a great deal on what country / language is your norm.

Which country/language has interesting IPv6-only content?

Your comment has been unfairly downvoted. There are IPv6-only users, but they can access IPv4 servers just fine through transition techniques. Otherwise they’d be unable to access most of the internet. You probably interpreted it as no IPv4 connectivity at all, which as you assumed is impossible.

If you only care about the CDN parts, you can change the security level in the Cloudflare firewall settings to Off and it won't serve any captchas.

HCaptcha is the absolute worst, their captchas are consistently harder, the datasets are impossible to decipher and the failure rate is much higher, even if I'm sure that I chose the right options. My parents had no chance last time they ran into one.

CF should have just rolled their own captcha service instead of buying into someone else's public ML training program again.

reCaptcha is definitely worse, but it goes easy on people logged into a google account.

Settings » Firewall » Settings » Security Level (Set to "Essentially Off")

These are the five SSL options for a Cloudflare website [0]:

1. No SSL: User <--HTTP--> Cloudflare <--HTTP--> Origin Server

2. Flexible SSL: User <--HTTPS--> Cloudflare <--HTTP--> Origin Server

3. Full SSL: User <--HTTPS--> Cloudflare <--HTTPS--> Origin Server; Self-signed cert ok, expired cert ok

4. Full SSL (strict): User <--HTTPS--> Cloudflare <--HTTPS--> Origin Server; Origin server must use an SSL certificate that Cloudflare provides [1]

5. Strict (SSL-Only Origin Pull): User <--HTTPS--> Cloudflare <--HTTPS--> Origin Server; same as Full SSL (strict), but you pay need to pay Cloudflare more money

---

3 and above will fix this issue as they encrypt from Cloudflare to the Origin Server.

This is the traffic flow from the link:

User -> Cloudflare -> Airtel -> GitHub Pages

Where the connection with flexible SSL is Cloudflare <--HTTP--> GitHub Pages.

Upgrading to Full SSL (or higher) and using HTTPS on GitHub [2] should fix.

---

Alternatively, deploy your static website with Cloudflare Pages [3], which has feature parity with Github Pages.

The flow would then be: User <--HTTPS--> Cloudflare Pages

[0]: https://developers.cloudflare.com/ssl/origin-configuration/s...

[1]: https://developers.cloudflare.com/ssl/origin-configuration/o...

[2]: https://docs.github.com/en/pages/getting-started-with-github...

[3]: https://pages.cloudflare.com/

EDIT: The replies by kentonv, x1110dc, and r1ch all have valid points.

> 5. Strict (SSL-Only Origin Pull): User <--HTTPS--> Cloudflare <--HTTPS--> Origin Server; same as Full SSL (strict), but you pay need to pay Cloudflare more money

The difference in this mode is that even if the client connects to Cloudflare using HTTP, Cloudflare will connect to the origin using HTTPS. In all other modes, if the client connects by HTTP, then Cloudflare will connect to origin by HTTP.

Of course, most people these days enable "HTTPS only", in which case Cloudflare will redirect HTTP clients to HTTPS and therefore not make any connection to the origin at all for HTTP clients.

For #4 (strict full) the origin server doesn’t have to use a Cloudflare-supplied cert. Any cert issued by a publicly trusted CA will do. See https://developers.cloudflare.com/ssl/origin-configuration/s...

Exactly, I've been doing this with letsencrypt.

How do you get certbot to work? Or do you renew manually or something?

I just went though this. you need to whitelist the acme-challenge (using page rules) like this:

  *example.com/.well-known/acme-challenge/*
  Disable Security, SSL: Off, Cache Level: Bypass, Automatic HTTPS Rewrites: Off

and one big gotcha:

  Under SSL/TLS -> Edge Certificates -> disable Always Use HTTPS

(assuming you are using the HTTP-01 challenge).

Getting it to work the first time was a pain. Basically, you want to disable cloudflare (just untick the box so that it goes directly to your server, you can keep using cloudflare's dns server), then obtain the normal way, and reactivate Cloudflare. But I would highly recommend using cerbot's cloduflare dns plugin[1] instead so that you can (re)create the certificate w/o disabling cloudflare.

1: https://certbot-dns-cloudflare.readthedocs.io/en/latest/

Note that while option 3 will fix this particular issue (because they only seem to care about port 80), it doesn't stop them from MITMing the connection with their own self-signed cert in the future. Only options 4 and 5 ensure a fully secure SSL connection.

Glad to see this getting attention. Flexible SSL is an awful option that has no place in the modern encrypted web. Out of the four SSL options Cloudflare gives users, only one is actually secure. It's a huge foot-gun.

I assume you mean only ` Full (strict)` is secure, and not even `Full`?

I happened to be recently looking at putting cloudflare in front of an S3 bucket, and it looked maybe easier/more feasible to do with `Full` instead of `Full (Strict)` -- because you can skip configuring the S3 bucket have an SSL cert for your actual front-facing domain (which can be cumbersome and/or more expensive to set up) and just let CloudFlare connect to it as `*.s3.aws.com` or whatever. As long as CloudFlare is actually requiring a trusted cert for *.s3.aws.com, are there security implications I'm missing for why this is a bad idea? (I guess AWS or someone with the keys to AWS certs could be spoofing you? Anything else?)

Or in general even without reference to this use case, explain vulnerability examples/threat models for `Full` without `Strict`?

Full does not check certificates, so a MITM can supply any untrusted certificate and intercept your traffic. Only Full (Strict) does trusted cert checking.

"Can" theoretically, or there's a valid attack vector which one may pull off immediately in the wild?

The current Airtel scenario is a good demonstration I’d say.

Cloudflare terminates SSL at the point closest to the user, so any of the ISPs between Cloudflare's POP and the origin could MITM the traffic. Normally an untrusted MITM cert would throw up a TLS error, but since the Cloudflare <-> End User part is trusted, a MITM attack with anything but Full (Strict) is undetectable to an end user.

There's plenty of examples of ISPs trying this in the wild: https://www.zdnet.com/article/kazakhstan-government-is-now-i... https://www.reddit.com/r/sysadmin/comments/4vy3op/my_isp_is_... https://superuser.com/questions/176651/isp-replaces-ssl-cert...

Huh, I didn't realize Full didn't check certs at all.

Why don't they have a mode that checks that the cert is valid for hostname that you have configured cloudflare to access (which *.s3.amazonaws.com already has), rather than for the end-point that the destination isn't actually answering on directly itself? Or maybe I'm misunderstanding what was up, have to mess with it more.

Just use a free Cloudflare Origin CA certificate and set to strict. Encryption for the whole round trip unless someone accesses origin server directly.

https://developers.cloudflare.com/ssl/origin-configuration/o...

But now do I have no way to access the S3 bucket not through cloudflare?

You know it's supposed to have a certificate from Cloudflare, so, you can choose to trust that certificate (knowing it is correct for this purpose) or you can use tooling that doesn't care.

If you mean can you present it for general access by people on the Internet, no, not really, because the certificate aren't trusted, you would need a proxy layer.

On the contrary, the idea that a mom-and-pop shop running an HTTP website on some ancient shared hosting can easily just drop in Cloudflare is a Good Thing. Sure, you're not removing the attack surface, a state actor could intercept the server-to-server connections... but at least the hacked router in your customers' coffee shop is removed from the threat model. You make it any tougher, and that HTTP website is just going to stay HTTP forever.

Hence, my recommendation of a "Big Red Warning" on picking anything lower than Strict SSL.

A user must knowingly choose this option, and many websites which picked Flexible SSL could easily be upgraded to Strict SSL today. CloudFlare must work towards upgrading these users, especially where it can easily identify that SSL is supported at the origin server (such as Shopify, GitHub etc).

The downside of such a setup is that as a user, I now have no idea if the site is actually secure. If the mom-and-pop shop is taking credit card orders and has a secure Cloudflare cert, how do I know they aren't then transmitting my credit card number and other personal data unencrypted the rest of the way? If the website was HTTP I would know as a user not to enter sensitive info.

This is why I think flexible SSL is worse than no SSL. Cloudflare's own docs used to say Flexible SSL "should only be used as a last resort if you are not able to setup SSL on your own web server, but it is less secure than any other option (even “Off”)" (this has since been removed).

> The downside of such a setup is that as a user, I now have no idea if the site is actually secure.

You never do. SSL is not sufficient for security. It protects against a single attack vector, that's if it's set up correctly. For all you know, the set up of the server your visiting might have be old enough to have a driver's license, with public SSH and the root password "12345".

Feedback: I might suggest changing "Now Resolved/ Not Resolved" to Resolved/Broken. This could help with readability as right now there is 1 char difference and it took me a few times to parse.

The column was quite misleading, as "resolved" doesn't mean that airtel is no longer trying to block it. (it's trying, but failing).

As such, I've just dropped the column.

Cloudflare powered censorship in Pakistan works in a similar fashion. ISPs block websites, but because Cloudflare's data center forwards using local ISPs, you get a nice secure blocked page.

I did come across several reports from Pakistani users as well while researching for this today.

Do these ISPs limit themselves to court-ordered blocks in your case?

Could you also mention Turkey & Pakistan in your letter as well?

Same here in Turkey. But this one is an int'l transit provider.

Posted this[0] months ago, even sent emails to Cloudflare NOC multiple times and nobody did even care.

[0] https://community.cloudflare.com/t/censored-pop-orpheus-not-...

Ages ago Cloudflare started to advice against these kinds of setups. I did not even know it was still possible. TLS termination on the edge services is just stupid.

Disagree. TLS terminated at the edge, by a trusted partner, is perfectly valid and in many cases a great plus. You should, however, then always have a tls connection back to the main host, this is obviously not always the case and is wrong. Ideally CloudFlare would not make it possible.

By terminating at the edge it enables many useful features of services such as CloudFlare that would otherwise not be possible such as the “web application firewall”.

If you think of CloudFlare as a hosting provider in the same way as any other (which they are) it ridiculous not to trust them terminating TLS.

TLS termination in the sense that you terminate TLS and then send the traffic to the origins without encryption. Cloudflare services basically works at layer 7 so they must decrypt all the traffic.

Ah, your original comment was not clear, we agree then.

I'm just frustrated by the bandwagoning criticism of any use of CloudFlare and the suggestion anyone using them is MITM their own visitors, when clearly they are just another part of your own infrastructure (when used correctly). Your comment "TLS termination on the edge services is just stupid" made me think you were doing that.

TLS termination on the edge services wasn’t stupid back when crappy vendors (Github Pages included) didn’t support TLS. GitHub Pages only added TLS for custom domains in 2018.

I also tried reporting this issue to Cloudflare in the past through their hackerone page (since I was out of ideas where to get the request though), this is the response I got:

> Enabling SSL/TLS between Cloudflare and the origin site is a customer decision. When this protection is not enabled, as is the case here, an ISP can manipulate the requests before they reach Cloudflare. If this behaviour is not desired, the customer must change the settings for the site in the Cloudflare dashboard.

The request in question gets MITMed after it reaches Cloudflare edge servers, the connection between browser and the edge server happens over SSL

In the early 2010s for a couple of years all Cloudflare websites would sometimes fail to load on my home connection - 20% or so of page loads would hit a strange error, some sort of tcp or ssl issue

It’s good to see others experience the downside of centralising the internet. Until reading this, it seemed like everyone blindly loves cloudflare.

Why not use Github Pages without Cloudflare? It also supports native IPv6 if that was the reason.

https://news.ycombinator.com/item?id=28710629

A lot of the impacted websites are old, from before GitHub Pages offered TLS support.

Cloudflare is not doing anything wrong, why to pick that call ?

Airtel support is here https://www.airtel.in/contact-us

Also, vote with your wallet and don't use Airtel ?

Even better: vote for a party that is not BJP? :)

Yes vote for congress.

Oh but they are. They chose Airtel in their locations. I'm using BSNL as my ISP but my request goes to one of these locations that seems to be using Airtel and I get the same DoT block message.

Users who are not Airtel customers are impacted.

What if the site owners change their SSL settings to Full SSL though?

It fixes the issue, and that has been the recommended fix I've been offering developers all this while.

So really website owners are just misusing Cloudflare? How is this Cloudflare's fault?

In this case, it is not a random party in the CloudFlare--GitHub connection (say a network operator in UK).

The snooping intermediary (Airtel) in this scenario is one that has a commercial relationship with CloudFlare and powers CloudFlare's network.

CloudFlare has been aware of this issue for years, but it hasn't done anything to get its vendor to fix their network.

Isn't the censorship applied by Airtel, Jio, etc, because of local laws?

https://en.wikipedia.org/wiki/Internet_censorship_in_India

I don't see how Cloudflare or any other provider can make Airtel "fix" the snooping when Airtel is forced by law to block those sites.

This seems to be a policy/government problem, not a Cloudflare or Airtel problem.

There is no court order to block neovim.io or teachyourselfcs.com.

Airtel isn’t forced to block these sites. It is blocking these because of a mis-configuration somewhere.

2 of the 4 SSL options Cloudflare provides will give a nice lock icon in your address bar and HTTPS will all look like it's working fine, but the connection to the origin is still insecure. It's very misleading and IMO irresponsible of Cloudflare to offer these options.

Well, they're sending packets to an upstream ISP that then does the wrong thing with them. They certainly have more leverage to get that upstream ISP to clean up its act than their customers do.

More broadly, if enabling a particular Cloudflare feature (in this case, Flexible SSL) constitutes "misusing Cloudflare", then Cloudflare should simply not offer that feature at all. There's a bit of a balance here; when they introduced it in 2011, a lot of hosts didn't offer HTTPS at all and none of them offered it for free. MITM is genuinely more likely to happen between the end user and Cloudflare than between Cloudflare and the origin server (because the former can involve things like unsecured coffee-shop wifi), so for webmasters who couldn't use end-to-end HTTPS, it provided a real security benefit—which had to be weighed against the cost of telling end users that their connection to the site is secure against interception, when that wasn't entirely true.

I think there's a case to be made that this tradeoff was worth it in 2011 but is not worth it in 2022; today, end-to-end HTTPS can be had for free, and is easy enough that there's usually no excuse not to.

Especially when CloudFlare knows that their upstream is specifically targeting a certain class of users, they have a responsibility to fix this and notify their impacted customers.

Full isn't actually enough to secure this - with their MITM position, Airtel could send an untrusted certificate and continue to intercept and modify traffic. Full (Strict) is the only safe option.

I read the page and there seems to be nothing to back up the claim that this vendor is "snoopy?"

Couldn't you just setup a proxy or VPN in the "cloud" and bypass all of this?

Actually, no. I run these tests on Digital Ocean servers in Bangalore, which almost always hits the DEL Cloudflare POP, where Airtel is the upstream and blocks the connection.

To fix this, an Indian user must actively avoid Cloudflare POPs, which is a bad proposition. Also, it shouldn’t be up to end-users of Neovim to fix what is CF/Airtel’s fault.

Are there alternatives to this service CF provides?

what are the use-cases for cloudflare in front of GitHub pages?

This is answered quite clearly in the linked page.

GitHub Pages originally didn't support SSL on custom domains, so people would often put Cloudflare in front of it.

Now GHP does support SSL on custom domains, so Cloudflare is no longer needed, but obviously a lot of sites still exist with the original setup.

> GitHub Pages originally didn't support SSL on custom domains, so people would often put Cloudflare in front of it.

I've done this myself and I'm fixing it now!

(Some JavaScript APIs I've experimented with over the years require HTTPS - from WebAuthn, which hey, fair enough - to Web MIDI, which hey, what the heck? https://developer.mozilla.org/en-US/docs/Web/Security/Secure... )

Even the webcam API requires HTTPS, I guess that makes sense?

If it didn't, then whenever you visited a non-HTTPS site, an MITM could inject code into the page that turns your webcam on and then spies on you. Permission prompts help somewhat with this, but not really enough, given that the address bar provides no clue that you're not talking to who you think you are (beyond the generic "Not secure" warning that users usually ignore since it's so common). Allowing this would be a total disaster for user privacy.

Ah, that makes a lot of sense! Hadn't even considered that, thanks!

Web MIDI isn't for playing .mid files in the browser. It's for letting in-browser JavaScript talk to arbitrary hardware devices that speak the MIDI protocol; this includes not only things like keyboards and synthesizers, but also stage lighting systems and other theatrical devices that use MIDI as a control protocol (so that you can have, e.g., lighting and sound cues happen in sync, because they're all centrally controlled by a single computer). In general, any API that controls hardware counts as security-sensitive; you don't want MITMs to be able to hijack your hardware devices and make them do something bad.

The major browser vendors committed to all genuinely new features being Secure context only. Secure context means either HTTPS or the local machine.

If the feature is something you could do with a polyfill anyway then it's not really a new feature, it usually makes no sense to restrict it. But something like Web MIDI you clearly can't build with a polyfill, so restricting that makes sense.

The reason to do this for new features is that by definition if your HTTP site already works, it didn't use them, if it relied on features that didn't exist it didn't work. So this prevents backsliding, your HTTPS site might stop working if you downgrade it to HTTP, but your 1997 HTTP site still works since obviously that doesn't use Web MIDI, or indeed probably CSS.

For a few features, vendors are going back and removing Insecure Context support, but this work is slow because inevitably even if it was a terrible idea people relied on it working (e.g. EME is used extensively from HTTP, because you know you really care that people can't pirate your mediocre training video, but you don't care enough to prevent them trivially intercepting it during download...)

GitHub Pages also doesn’t (yet) support custom headers and you can add them with Cf via Workers. So if you’re concerned about the results of securityheaders.io, for example, you can add those in.

GitHub has a traffic limit. If you serve mostly static data, Cloudflare can cache most of it, so you stay within GitHub limits.

I ran into the same issue a few weeks ago. https://news.ycombinator.com/item?id=29685966

Poor Snoopy, after a terrible experience at the supermarket, now this! :-(

https://www.youtube.com/watch?v=nD2n7Jiy-xY

Downvoted because of a simple and inoffensive joke that relates to Peanut' Snoopy character. The lords of seriousness don't tolerate humor.

Reply on news.ycombinator.com