When the policeman becomes the criminal – how Cloudflare attacks my machines | Hacker News Reader

When the policeman becomes the criminal – how Cloudflare attacks my machines

kmitov.com

44 pointscoolgoose30 comments

Okay, so i have multiple problems with this blog post. I want to give the author the benefit of the doubt. They're a CTO at some startup, surely they know how cloudflare works. But i dug a bit and i don't see evidence of this.

In short, for the non-cloudflare-users among the HN crowd, and since this is a super clickbaity title, cloudflare is a reverse proxy. ALL IPs in the user's access.log, unless explicitly configured otherwise, will be cloudflare's.

Cloudflare also doesn't proactively update its cache. It does so on demand.

What I'm really confused with is if they did indeed talk to CF support, how was this not caught by the support agents? Could they genuinely be CF IPs? Having a hard time believing that.

Anyway, first step would be to look at the x-forwarded-for header. If not available, since these are logs, cloudflare has traffic analytics and wouldn't confuse its own IPs in the mix.

Their previous detailing of GoDaddy/Microsoft issues has the same sort of "it's someone else's fault!" but reads heavily like a misunderstanding on the poster's part.

https://kmitov.com/posts/godaddymicrosoft-365-and-how-an-ema...

(In this case, that GoDaddy resells Office 365, and the interactions between the two.)

OP is very misguided. Cloudflare does not proactively refresh their caches. That's just not how CDNs work.

Here is how I would troubleshoot:

1. Look for the X-Forwarded-For header: All requests or the AWS server will come from Cloudflare. That's the point of a CDN. The X-Forwarded-For header tells you the actual client IP making the request. I think Cloudflare has some tools to block certain IP addresses. You can use that once you have identified the IP to block.

2. Make sure nginx returns appropriate headers so that Cloudflare will cache the responses from the origin server. If caching is set up correctly, Cloudflare will not make requests to your server. They will serve from their cache instead.

It really sounds to me like someone is just spidering his site each night using curl and it's causing a cache refresh. He doesn't go into detail about the "547 hosts" to explain if they're all fetching the same content or if the whole site is just being fetched from a cluster of machines as the requests arrive at Cloudflare. But either way it sounds like he's misunderstanding how Cloudflare works.

I've run into a very similar issue, but it was bots - not the CDN provider.

Every day (to this day) we're getting thousands of requests for images that no longer exist on our CDN (because they were stale/deleted). The CDN normally does not hit the origin machine (where the images are hosted) unless it cannot find the images on the CDN, at which point it queries the origin for the image. Problem was, the image no longer existed on the origin. I didn't expect the origin would receive much traffic, but suddenly it's receiving a ton of traffic.

I was very confused because, at first glance, it looked like I was being attacked by my own CDN provider given the tremendous traffic and the fact that the CDN provider was the only thing allowed to access that box (the origin).

At any rate, I contacted the CDN provider and informed them that thousands of requests that resulted in 404's were taking down my website. They told me there was nothing they could do.

In any case, I managed to wrangle together some new infra to handle it. I don't think whoever was hitting the CDN for those images was malicious. However, it occurred to me that had they been malicious, then they could have just hit random non-existent file-names at a much higher rate and done a lot more damage.

I have to give credit where it's due - you can actually talk with humans at Cloudflare, unlike Amazon, Google, Microsoft or other big companies.

However, given how long it took to get an answer to a very simple, basic question about reported abuse, I don't know if you'll get a satisfactory answer about why this is happening.

Please do report back if you do find out more.

You can talk to humans at Microsoft and Amazon for very reasonable support rates. Google…

Google Cloud also offers (paid) support: https://cloud.google.com/support

Human on a phone for technical issues starts at $500/month + 3% net spend though.

What needs to be enabled is what is typically called "origin shield" - this will make your CDN's edge servers use their own internal cache instead of refreshing everything from the source.

Author may also want to review cache headers returned for their content as it should be possible to cache at CDN effectively indefinitely.

Kind of an overly-sensational title, but it sounds like a legitimately frustrating situation to be in, and.. I guess I'm not surprised? It sounds like support in this situation has little incentive to believe the fault is on their end, but I think we all know if this blows up on social media then this ticket will receive higher priority.

The article is a little light on data though. No vertical axis, no information on the volume of data transferred or the associated costs. I feel like its pertinent information, and if the numbers are small enough that it makes the title seem silly well.. dial it down a little, I guess.

A solution could be to use Cloudflare Worker scripts. You would store valid file names in their kv storage and then check if the incoming request is valid before forwarding it. This way you wouldn’t need to spend more money on infrastructure. Another good solution is to use Cloudflare’s advanced rate limiting rules. You can block users who hit too many 404 requests within a certain time frame. But this requires a business or enterprise plan.

Spending more money on increasing infrastructure rather than reading through Cloudflare’s documentation and offerings seems kind of a waste…

OP is seeing the proxied requests but looking at the wrong data. Should check X-FORWADED-FOR and create rules to stop it using the WAF provided. There is no "Cloudflare attack" going on.

I have some machines on Digital Ocean and asked if I can set a global spending limit for my account. The answer "No. Digital ocean does not provide it

This surprises me. I thought the only thing DO said no to - is anything at all that might protect us from malicious, DO hosted traffic.

Since I see a lot of OP, I have no idea who the guy is, just found it interesting mostly because aws and digital Ocean don't allow payment caps, but gcp does for eg

How's that an attack, really?

TLDR it's not an attack, CF isn't a criminal, he's a CF customer and sees traffic spikes with a curl useragent.

Yeah, I'm not seeing much evidence of the allegations in this article. CloudFlare would be very, very stupid to have set up a cache stampede of the sort OP describes, and they don't use cURL as the user agent (https://support.cloudflare.com/hc/en-us/articles/200172556-W...).

I kinda wonder if someone's using https://workers.cloudflare.com/ to crawl their site daily. Or they just need to configure `ngx_http_realip_module` to get the originating user's IP address, if they're not currently processing the X-Forwarded-For header.

You left out the part where the requests are coming from Cloudflare IPs.

Cloudflare is a reverse proxy so all requests would be coming from cloudflare. Did the author look at the x-forwarded-for IPs?

It really depends on whether all traffic from cloudflare IP's or just the most frequent ones. If the former, then they have configured nginx wrong and are logging the wrong source ip.

I don't see evidence they aren't logging the wrong source IP (which is the default format with nginx, which the author is using).

These are extraordinary claims, which require extraordinary evidence.

Typically when you're using Cloudflare you configure your origin firewall to drop traffic that doesn't come from Cloudflare [1]. What would be the point of using Cloudflare if you also accept traffic that bypasses Cloudflare? How would you even set up the DNS for that?

1. https://www.cloudflare.com/ips/

No. If you're a CF customer, all traffic comes from Cloudflare IPs, that's how it works, they're providing a reverse proxy for you.

> sees traffic spikes with a curl useragent

And from Cloudflare IPs.

That's how Cloudflare works. Traffic goes user --> Cloudflare --> your server, with an X-Forwarded-For header.

The same happens with most load balancers. In Apache, you have to set up mod_rpaf or mod_remoteip. In Nginx, you have to configure ngx_http_realip_module.

Yeah, I get that, but I'm replying to someone who doesn't seem to think the traffic came from Cloudflare.

Hmm.. this seems to be about a few thousand requests per day that CF is doing.

How much can those cost him? Not more than a few pennies per month, right?

He could just stop using CloudFlare if he does not like it.

Calling it an "attack" is a bit much.

One of the example requests is to a 27MB .webm file; videos could get pretty expensive with AWS bandwidth charges.

Oh, I see.

He says it's hundreds of gigabytes of traffic. I don't know the prices but sounds more than a few pennies

Reply on news.ycombinator.com