Hacker News Reader

A couple other notes:

* The IP address 2a06:98c0:3600::103, mentioned in the article as appearing in the CF-Connecting-IP header, is a special IP address that is used for all cross-zone requests that come from Workers. This is not the actual address of any Cloudflare machine. Workers fundamentally don't have distinct IP addresses. Instead, the CF-Worker header needs to be used to identify which worker sent the request. We intentionally do not identify the original client's IP here because the Worker itself could be working against the client, and could have maliciously modified their request to be something entirely different. Hence, the request can't be considered to have come from the original client.

* Any cloud service that lets you make HTTP requests can, obviously, be used to set up a proxy that hides your IP address. Workers is actually less useful for this compared to many other cloud hosts since we tag all outgoing HTTP requests from a Worker with the sender's domain name (in the CF-Worker header), which makes it relatively easier to track, report, and block abuse.

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name.

Out of interest do you provide any automatic checks for this? It seems like it would be trivial to add some kind of verification step that simulates a request with a bogus host header and checks the status code of the response.

What would CloudFlare automatically check the Host header against? The Host header would be set to the attacker's domain (which is in CloudFlare) and, per OP, CloudFlare does not verify ownership of the origin server IP.

It would be a nice feature if CloudFlare customers could opt-in for origin server IP verification. Once the origin IP is verified, it would prevent other CloudFlare users from using the IP as their own origin. However, I understand why CloudFlare doesn't verify origin IPs -- it would require a verification process that doesn't rely solely on uploading a verification file to a web accessible directory. It'd be a bit difficult to keep the verification process simple for end-users if they aren't running a web server. Or maybe I'm wrong and non-HTTP use cases don't need to be considered and a verification file uploaded to a web accessible folder would work fine.

Somewhere in the interface you’d have a button that when pressed would send a request to the specified origin server with a Host header set to “foobar.com”. If the status code is 200 you’d display a warning saying “perhaps you should re-read our documentation about verifying host headers”.

I can think of a whole bunch of corner cases to this, but that’s the general jist.

But in order to safely enable that check for all requests that go from CloudFlare to the origin IP, you would first need to verify ownership of the origin IP.

You wouldn't want users who don't own the origin IP to be able to turn on that sort of check and potentially cause a service disruption. It could be the case that the legitimate owner already has a Host header mismatch in their HTTP requests/responses.

Presenting a warning about possible misconfiguration to the customer need not create a service disruption.

We're talking about a situation where an attacker would create the service disruption by enabling Host header checks on an origin server that the attacker doesn't own. So a warning wouldn't help because that warning would be displayed to the attacker.

The logic is tricky, but in the end origin server ownership verification is what's required to safely support Host header checks.

If the attacker is logged into the Cloudflare control panel, where presumably the warning would be displayed — where else could it be displayed? certainly not on the content served to end users — then, yes, the attacker could clear the warning. They could also change the origin servers, or do countless other things to disrupt service, that do not require modifying an origin server. I consider that an acceptable failure case for the warning.

I'm not arguing for or against ownership verification, but there is opportunity to improve here that does not depend on the question of ownership verification.

I could understand verifying domain names, but verifying IP addresses seems like it would be an extremely complex feature for very little benefit.

You'd have to have implement it in-line with the dns resolution in the server before talking to the origin, right? There's no other time you could reasonably do it (DNS changes), and it would have to be checked on all customers requests, not just the ones that opted into the feature (because those other customers could be the "attacker"!). Of course, you would cache it and use an efficient lookup scheme, but in general cloud services really don't like creating features that have non-localized effects like this.

Not to mention this sounds like a support nightmare foot gun for enterprise customers.

I don't think we do, but that's a good idea! I think it could be automatic. We'd have to first detect that the origin server appears to be blocking non-Cloudflare IPs, by seeing if it responds to requests from some random non-CF IP address. Then we'd verify that it refuses to respond to a request for some dummy domain. If it serves the site's normal content when the host name is from the wrong domain, then we could alert the customer that they seem to have an insecure configuration. I will suggest this internally...

The “normal content” bit is possibly quite hard to detect. You could do some kind of distance calculation between “safe” responses and “unsafe responses”, but it seems like there would be a bunch of edge cases given the variety of sites you host.

Anyway, thanks for hearing my suggestion!

Yeah, I think I'd just look for HTTP status code.

Assuming you're using HTTPS (and if not, why?!), this is checked automatically as part of the TLS handshake. The only downside is Cloudflare makes it easy to accidentally disable this check as "Full SSL" bypasses certificate verification.

Yeah, I do feel like there should be more visible encouragement to use "Full (Strict)" when someone adds a new domain for the first time. Especially now that it's so easy to provision free SSL certificates via Let's Encrypt etc.

It seems to me that author of the article implies that Cloudflare treats Workers' IPs as having much higher reputation (threat score in CF's terms) than any other IP on the Internet by default. This allows worker to bypass bot protection as it does not trigger at all or only when zone is configured to trigger on all requests (under attack mode).

Could you comment on that?

It looks to me like it’s just about how you can use Cloudflare as a reverse proxy, to hide the fact that your origin is Tor. And the bit about being able to change X-FORWARDED-FOR seems to just be about defeating IP-based rate limiting on sites that trust Cloudflare sufficiently to respect the X-FORWARDED-FOR header.

To be honest, I don't know anything about Cloudflare's bot management product. That isn't my department.

I would speculate that proxying through any cloud service (Workers, AWS, Digital Ocean, whatever) can indeed be a tool to hide bot activity, and that any bots product needs to think about how to combat that. I would also speculate that originating IP address is not the only signal the product uses to determine threat. Moreover, I would speculate that this is all an arms race, there will be a constant stream of new techniques developed by bot authors and new techniques deployed to thwart them.

You may be right that the blog author intended to report a problem specifically in bot management, and I have no idea whether that specific problem is real. However, the blog post is not very clear, and many readers have misinterpreted it as reporting a way to totally bypass all Cloudflare protections, such as WAF, geo firewall, rate limiting, etc. That's the misunderstanding I meant to address. Sorry, I could probably have been clearer about that.

>If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name. If they do not verify the Host header, then anyone can sign up for Cloudflare and simply configure their DNS to point to the victim's origin IP address, and requests will be routed there -- but will have the attacker's domain in the Host header.

What exactly could such a misconfiguration enable? Just that the attacker could configure lower levels of Cloudflare security settings for their domain, to bypass the origin's intended Cloudflare security settings? Would this also bypass I'm Under Attack Mode?

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name.

Where is this documented?

I'm 100% sure it isn't. Cloudflare docs are not great, and as a Enterprise customer I spend more time with support than needed as a result.

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name.

The worker code sends the correct host name.

https://github.com/jychp/cloudflare-bypass/blob/dd7c1cfc2e6b...

Right. If the hostname requested by the worker is a host that is itself protected by Cloudflare, then all of that host's usual security settings apply. Nothing is bypassed.

The author of the blog post seems to have been confused by the fact that different behavior happens if the target host is on Cloudflare vs. not on Cloudflare.

> any requests coming from a Worker will have the CF-Worker header identifying the zone which sent the request. If a customer suspects abusive requests coming from Workers, they should report it to us.

What happens if you receive a report of abusive behavior from a Worker? Would you rate limit the specific Worker, or would there be any impact on the larger Cloudflare account of the Worker's owner?

For instance, if we have a Worker that connects to an arbitrary endpoint based on user-generated parameters, or some vulnerability is found (in the JS code for the Worker) that allows this to happen, and it ends up being used for abusive behavior, would we have recourse to fix the vulnerability, or would action be taken immediately? Would our larger account, or our unrelated Workers, be at risk of being taken offline?

These issues are handled on a case-by-case basis. I can't make any hard promises here (it's not my department), but generally, if your worker is not obviously malicious, we will talk to you before taking any drastic action.

> If a Cloudflare customer has configured their origin server to respond only to Cloudflare IPs, then they MUST also verify that the "Host" header on any request actually matches their domain name.

This could be improved considerably if Cloudflare removed support for insecure TLS options. "Full" SSL bypasses certificate hostname verification on the origin server which could prevent such an attack at the TLS level. Every time I've added a new zone to Cloudflare, Full or Flexible (ie, no SSL) has been the default. The only safe option is Full (Strict) but I'd bet good money that the majority of Cloudflare customers aren't using that.

This seems like the same kind of reason we're not using Cloudflare Access to protect some internal resources.

We'd love to use Argo Tunnel but unfortunately it breaks with an OPTIONS request for us. We've tried opening tickets with Cloudflare.

Once upon a time: [link redacted]

Yep, I worked on that one. Thank you for reporting it.

Ironically, the purported problem in this case is the opposite: Because we don't attribute the request as coming from the original client (in order to avoid the security problem you reported), a client that has bad reputation associated with their IP can proxy through a worker to hide that. The answer is to assign reputation to the Worker itself.

Yeah, I don't envy you. Finding loopholes is more fun than trying to account for everything upfront.

You see the request on “both sides” of the worker, so shouldn’t you be able to evaluate the “connecting client to the worker’s reputation” when allowing or denying the upstream request?

You mean by somehow evaluating whether the incoming request and the outgoing one are sufficiently "the same", in which case the outgoing request can be attributed to the original client?

This seems hard. Let's say the request is the very simplest "GET /" request. The worker does nothing except rewrite the hostname, in order to proxy it to a different server. Is it "the same" request?

Well, imagine the new destination host is a company-internal dashboard that authenticates users based on IP address. Now this "GET /" request which was proxied will receive a response that contains secrets. The worker that did the proxying can intercept the response and learn the secrets.

So clearly if the request's host is rewritten, it cannot be "the same" request and it cannot be considered to have come from the original client.

But the only case where we care about any of this is when a worker makes a request to a different domain. If a request is to the same domain, we trust it, because in that case the attacker can only attack themselves anyway. So the only case that even matters here is when the host has been rewritten, and in that case, we've established that the new request definitely cannot be attributed to the original client.

So it seems to me there's nothing interesting we can do here. A request coming out of a worker (going to a different domain) must always be attributed to the worker itself, not to the client that triggered the worker.

From the article:

> X-FORWARDED-FOR: 1.2.3.4 (yes, we can override this header with whatever we want !)

Why aren't you guys disallowing that field from having any external value?

Because it's legitimate to have values before the request gets to CloudFlare, for instance if it was routed out through a corporate proxy, etc..

Ouch. I wonder if it's time to cryptographically sign those values or add a checksum or something.

Basically this is why you want to go full zero trust: if you use mutual authentication over TLS, this becomes much less of a concern since you’re not trying to craft policies based on IP addresses with multiple intermediaries.

Only possible if non-TLS or the company is man in the middling TLS (and certificate pinning isn’t enabled).

So... completely possible then.

So we're supposed to be appending the client's IP to whatever the client sent for X-Forwarded-For, documented here:

https://support.cloudflare.com/hc/en-us/articles/200170986-H...

For better or worse, this is how X-Forwarded-For works, by convention: each forwarder is supposed to append a value. This means that, even in a correct implementation, an attacker can specify an arbitrary prefix of values. The consumer must be very careful about which part of the chain they trust.

But the report here seems to be suggesting that we're mishandling this appending in some cases. We're investigating and if true will fix ASAP.

Because X-Forwarded-For is surprisingly complicated, we recommend using CF-Connecting-IP instead. As the docs say: "To restore original visitor IP addresses at your origin web server, Cloudflare recommends your logs or applications look at CF-Connecting-IP or True-Client-IP instead of X-Forwarded-For since CF-Connecting-IP and True-Client-IP have a consistent format containing only one IP."

> This attack has always been possible and is common to basically all CDNs.

I strongly recommend you play around with Azure CDN, because this is not entirely true.