Hacker News Reader

Comment by fivesigma | Hacker News Reader

Cloudflare is a major source of centralization.

The more providers offer something like this, the merrier. I understand that this isn't a layer 7 solution, but that has it's downsides as well - Cloudflare (or any other reverse proxy) will MITM all your TLS traffic, for example.

It's also time to address the elephant in the room: AWS. "Oops, you got DDOS'ed? Here, have a $50k invoice"

View on HN

[disclaimer: I work for CF]

CloudFlare also regularly speaks about attacks and mitigations, therefore is helping the community to build better defences. Other providers stay shy and never disclose their magic. We believe DDoS is an internet wide problem and one of the ways to solve it is to spread the mitigation know how.

Examples:

- DNS attacks https://www.youtube.com/watch?v=UcAygzNSxlI&t=2h13m20s

- Iptables is great https://www.youtube.com/watch?v=pCVTEx1ouyk

- Our DDoS mitigation pipeline https://www.youtube.com/watch?v=XiK4643YdOk

- BPF for DNS https://blog.cloudflare.com/introducing-the-bpf-tools/

- BPF for SYN https://blog.cloudflare.com/introducing-the-p0f-bpf-compiler...

- Kernel bypass with netmap https://blog.cloudflare.com/single-rx-queue-kernel-bypass-wi...

- NTP attacks https://blog.cloudflare.com/technical-details-behind-a-400gb...

- DNS amplification https://blog.cloudflare.com/deep-inside-a-dns-amplification-...

- Recent attack trends https://blog.cloudflare.com/a-winter-of-400gbps-weekend-ddos...

- L7 attack with ad networks https://blog.cloudflare.com/mobile-ad-networks-as-ddos-vecto...

Cloudflare gets so much undeserved hate on HN. You guys do amazing work, provide an incredible service and your writeups are awesome. Thank you.

I think their work is fantastic, but I really don't like that every site I visit meets me with CF's captcha page. It's not only annoying per se, but the privacy implications are unsettling. I welcome the competition.

This was never the intention. Part of the problem is inertion - cf operates large and complex application that was designed back when we had only a handful of customers. Part of the problem is technical - the privacy-centric anti-abuse technologies don't exist yet.

Please do help us fix this. Report issues, help us understand when we have incorrect IP reputation. Help us find captcha accessibility problems. And maybe - join the team to actually code the fix.

Problem is, your "incorrect IP reputation" concept is fundamentally flawed. As an example, I noticed that most VPN exit nodes have "incorrect IP reputation", which means if I want to browse the internet without my government spying on me, I have to wade through all your CAPTCHAs.

“fundamentally flawed” is not synonymous with “not supporting the style of anonymity which I prefer”. There's no evidence supporting the assumption that those VPN exit nodes’ IP reputation is actually incorrect rather than earned by the behaviour of other customers.

We went through this in the 90s where a few people were upset that they couldn't send email directly from their dialup connection, because they were still thinking of the world as it was in 1993 before spam became so prevalent and anyone who ran a mail server was constantly trying to deal with thousands of dialup IPs trying to deliver spam. What actually worked was that people changed the way they worked to use things like authenticated SMTP relays so you could simply block entire dialup ranges rather than try to come up with a spam-blocking AI.

The browsing system could be improved by something like a CloudFlare login system or long-term persistent authentication storage so you'd only see a CAPTCHA once a week. Unfortunately, most of the complaints come from very pro-anonymity users – which is a legitimate position but also means that it's a community which is going to be significantly more likely than average to have things like cookie & JavaScript blocking, so the complaints would simply shift to “I shouldn't have to create an account!” or “Why can't I disable JavaScript and cookies for your site!?!”

I stand by my assertion — the idea that an IP address can have "reputation" is fundamentally flawed. The SMTP example that you cite is actually a good one: the whole reason why we started assigning "reputation" to IP addresses is because we could not be bothered to improve SMTP over the last 40 years or so and it still assumes we live in a world of trust.

CloudFlare (and everyone else for that matter) should stop assigning "reputation" to IP addresses. An IP address is a network location, think of it as a temporary storage box which could be occupied by anyone and anything. We should move beyond coloring boxes.

So what's your alternative? People use addresses because they're a strong signal with relatively low false-positives (not many people are tor/VPN users with cookie blocking). How do we do better short of deploying a login system or a unique client identifier which cannot be disabled?

Keep in mind, unless you're using your own personal VPN off a self-hosted machine, you're likely to be sharing your IP address with other (potentially) malicious actors trying to hide their tracks

Same as if being on an ISP though.

ISPs have a much larger pool of addresses, so less or no sharing.

This is an inherent problem with numerous different people using the same IP addresses (VPN, Tor, etc). And might be a bigger problem than Cloudflare can resolve on their own. Cloudflare has to protect its network and its customers like a normal business, so VPN and Tor IP addresses will naturally be viewed with suspicion.

Also, I think the captcha solution is better than what most administrators do, which is to block those IPs outright.

There is probably no solution to this problem. Criminals regularly use those same services you're using.

Cloudflare's job is to block potential attacks, and blocking or CAPTCHA-restricting a subnet tied to a large number of attacks against their infrastructure seems reasonable.

Either switch VPN providers, or deal with the CAPTCHAs.

I love CloudFlare, but understand the concern some people have.

With the amount of sites they have behind them, and the amount of visibility and data they can extract, it is actually a bit scary to think what will happen if they get compromised or "do evil".

But so far, they seem to take the open and ethical approach and actually follow the "do no evil" mantra. CloudFlare is another Google in the making and I hope they keep like that.

thanks,

Yeah, but they've also the "Wifi Captive Portal" of the internet, which frequently creates trouble (if you're using Tor for example).

As a site operator you basically must use Cloudflare and highly aggressive policies regarding Tor.

I like Tor as a concept, the problem is that there is no way to stop people massively abusing Tor.

Whatever you want - nazi/hate speech, swatting, trolling, DDoS (by hitting expensive render paths or by forcing cache bypasses) - operate any kind of site with user interactions and you will get messed around with by mostly Tor-using scum, since trolls have understood by now that exposing their real IP will lead to them being v&d.

I don't like Cloudflare as a SPOF for half the internet, but for now they seem to be the only one able to reduce the impact of crap you have to deal with as a site op to a manageable level.

Trolls are destroying the Internet.

As a site operator, I have Tor (T1) whitelisted on all of my CloudFlare websites and have no additional spam problems or anything else. CloudFlare doesn't even catch all of my malicious traffic (spammers, mostly) -- even StopForumSpam doesn't do a 100% perfect job.

Counter-anecdote: I run a fairly small community and deal with an absurd amount of problems (spam, child pornography, threats, attempted attacks) sourcing from Tor and VPNs like PrivateInternetAccess.

I'm very glad CF exists. It certainly doesn't catch all bad traffic in my case either, but it helps. I don't think any cloud security service can ever stop all spam or bot activity.

> Trolls are destroying the Internet.

{blinking text|viruses|spam|IE6|blackouts|myspace|visual basic|2400bps|java|javascript|EMP}

rumors of its imminent demise might be exaggerated.. ;)

Tor is used for abuse. If you want the privacy* you get with Tor the price you pay is that sites and services will check that you're not one of the abusers. At least it's not a flat-out ban. Any other large network on such a small IP space (as an example maybe a NAT-using mobile ISP) would be checked just as harshly.

* Or any other reason you're using it.

Edit: If I'm wrong can you please reply explaining why? I'm having an emotional rollercoaster of up and downvotes here! Please do feel free to correct me if I'm wrong, we all need learnin'

that's a pretty poor excuse. why is privacy only available by submitting to a poor experience? privacy should be default, not a punishment

You're looking at it purely from the user side. Look at it from the admin side. Tor is basically a massive open proxy, and by blocking it or throwing up human checks like captchas, you eliminate a significant source of spam and abuse.

There's not much to be done about this otherwise - a Tor user is sharing a network with a significantly higher than usual amount of the bad elements of the internet.

You can put captchas where they belong—on comments etc. Instead, Cloudflare punishes people simply browsing; they nuke the 99.99999% of visitors that have zero intention of interacting with the page beyond doing few GETs.

Oh and don’t forget they’ll even put the captchas on subdomains, like img.domain.tld. Go visit stackoverflow via VPN/TOR and watch how the site has no styling/images even if you do their stupid captcha on stackoverflow.com. They’ll still serve you another one on static{1-50}.stackoverflow.com. Fun.

A few GETs spread out the right way and repeated often enough is a DDOS.

There's a very good reason why Cloudflare does this. Cloudflare isn't "punishing" anybody, their job is to protect the people that use their service. Tor (or any other open proxy) is a massive source of bogus and/or abusive traffic.

Of course they’re punishing legitimate visitors. If there weren’t any humans getting captchas there wouldn’t be any need for captchas in the first place.

It wouldn’t be so bad, except Cloudflare doesn’t have any idea what they’re doing. Like the static.domain.tld issue I’ve mentioned—not even documented. This breaks websites in a very, very, bad way.

Next we have the lovely security setting called “Essentially-off.” Which does absolutely nothing to the captcha system. Apparently “essentially” means “not really,” according to Cloudflare.

Also forget VPN/Tor. I’m getting consistently captchad by Cloudflare just visiting mostly american-only websites from residential networks in Europe.

They’re either incompetent or just plain malicious. Right now I’m swaying towards the latter :/

I run a site (not behind CF), and Tor is used heavily to scrape our uncacheable and extremely heavy to render pages for abusive/content theft purposes.

All you just said is that your site is broken.

Using a massively shared IP space of any kind is only available by submitting to a poor experience.

Nobody is saying "hey this guy has privacy, lets make this experience a pain in the arse". They're saying "Man this set of specific IPs are really hammering my system looking for WordPress exploits and we're not even running that".

To treat Tor better than regular traffic would be to discriminate against Tor, which would invite more abuse of Tor.

So carrier-grade NATs should also be captchad to hell. Somehow, they aren’t. Very mysterious, indeed.

Pure rough estimates, but consumer ISP subnets are probably ~1-3% malicious, while with Tor's rotating IPs, if you look over the past 5 years odds are at least 40% of exit node IPs have been associated with malicious activity at least once.

It makes perfect sense to add additional verification challenges for Tor users.

Also, anecdotal, but I work in infosec for a large US company and see a lot of traffic and read a lot of logs every day. The majority of all spam, fraud, and scan activity comes from Tor, or consumer-advertised VPN services, or various European VPSs. Residential ISPs (excluding Chinese and Russian ones) make up almost no part of it, relative to those others.

I'm pretty sure Cloudflare's challenge criteria is sourced from real data they've generated from traffic they've seen. They aren't trying to single out Tor users.

> It makes perfect sense to add additional verification challenges for Tor users.

I’m actually perfectly fine with that. The issue is how Cloudflare does it. They should just offer something like X-Cloudflare-Fraud-Score header or even <x-cloudflare-protected-frame /> tag (they’re already doing SSI), but instead they just nuke the hell out of everything.

The way they do ‘protection’ is openly hostile to open web, or even neutral web. Whether this affects Tor disproportionately or not is a minor point.

Do many malicious entities have easy access to carrier-grade NATs?

Man just read up I'm essentially just summarising this anyway: https://blog.cloudflare.com/the-trouble-with-tor/

Sure they do. They’re called ‘botnets.’[1]

[1] https://luminati.io/

I work for a hosting company. Can confirm that CloudFlare's posts help us building our own defenses and I'm grateful that they're sharing their knowledge.

So, thank you!

Calling this "the elephant in the room" is very insightful. It surprises me that AWS is so often recommended as the best choice for new businesses without large capital. AWS automatically scalable infrastructure changes DoS outcomes from 'your server is down due to DoS', to 'your company is out of business due to DoS' (and for small company it doesn't even need to be a large DDoS, shell script on a single machine with decent link can generate substantial AWS cost).

I've seen people state, in comments here and in blog posts, about DDoS attacks and also the resulting charge from Amazon - but in every case that I've seen the person goes on to say 'after talking with Amazon they dropped this charge'. Do you have any examples of companies or individuals being help liable by Amazon for these costs? Totally prepared to accept I'm wrong, just going on my experience so far.

> but in every case that I've seen the person goes on to say 'after talking with Amazon they dropped this charge'.

Negotiating the bill afterwards is an experience most people want to avoid, I think. I guess many would prefer a service where they don't receive the huge bill to begin with, even with other downsides.

Unless there is a contractual guarantee somewhere, that's not something I like to bet my company on.

Calling a reverse proxy company "MITM" seems kind of silly to me. It's not an attack, it's a service provider doing what their customers ask them (pay them) to do.

The letters stand for "man in the middle" which seems to me a quite literal description of what's going on. The word attack isn't used.

I know what it stands for and it is the name of an attack. Try Googling "man in the middle" and see what all the results say. It is not a neutral term.

When a service provider decrypts and filters traffic for you, it's usually just called decrypting and filtering traffic.

Except MiTM is used primarily, if not exclusively, in attack scenarios. Otherwise any third-party you use as a website owner is a MiTM.

If they’re stripping encryption and reapplying they’re MITM. Seems pretty straightforward.

It's amazing how you can redeclare the meaning of words and distort reality just to fit your narrative.

Stripping encryption implies that there was, at some point, some sort of encryption between the browser and the website itself, which there never ever was. In the Cloudflare architecture the browser never ever interacts with the website, always with the Cloudflare servers. When you've got the small lock, it never implied that you were connecting to the website, but always that you were connecting to what lies behind the domain name, which is the Cloudflare servers. As an additional point, the website owner has always agreed to use this architecture; they're not somehow victim of some kind of attack from a spooky middleman. They ask (heck, some of them even pay) for it. It is part of the way they want to serve content.

You should tone down the sentimental analysis and keep to the facts. Cloudflare is a reverse proxy service the website owner uses; it is part of their infrastructure.

> In the Cloudflare architecture the browser never ever interacts with the website, always with the Cloudflare servers.

Indeed.

It's almost as if metaphorically speaking they were standing there between them... like a man... in the middle...

The name on the certificate is not Cloudflare’s. It’s origin’s. There’s no legitimate mechanism for reverse proxies to operate HTTPs sites, other than to act as a dumb pipe. Calling it MITM is therefore entirely appropriate. Whether this protocol abuse is sanctioned by origin or not is entirely beside the point.

Simply decrypting and re-encrypting web traffic in an intentional and authorized way is not MITM and should not called that.

So, why do people call CloudFlare MITM?

1) They honestly don't know that MITM is just the name for an attack, not a neutral term.

2) They think any mid-route decryption/re-encryption is bad and dangerous.

3) They are mad about some aspect of CloudFlare's service, and call it MITM to make it sound worse to everyone else.

Folks in the 1 and 2 camp just need to get educated. Folks in the 3 camp might have good points on the substance, but they are contributing to the misunderstandings of 1 and 2.

Assumption: End-to-end encryption to the name on the certificate.

Reality: Someone else (Cloudflare) claims to be the name on the certificate, whereas in reality they’re not.

That’s the textbook definition of MITM attack, whether you like it or not.

Of course all CDNs work like that. No one’s claiming otherwise. Cloudflare is just bringing all the ways internet is fundamentally broken to the forefront. Their unapologetic and extensive abuse of those design flaws (captcha pages) are a constant reminder of just how a hopeless mess the internet is.

And where is the website hosted? If you're on AWS, guess what, Amazon is decrypting the traffic, not the website owner.

Calling Cloudflare MITM but not Amazon is just demonstrating that you haven't thought carefully about how professional service providers work together to deliver a website.

The certificate authenticates that you are seeing the website that the website owner intended you to see. If the website owner hires Cloudflare as a professional service provider, that premise is not violated!

The difference is Cloudflare is perfectly content to forward the backbone traffic unencrypted over public internet. In fact, they advertise this malfeature.

Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here. There’s plenty of better reasons to hate Cloudflare anyway.

> The difference is Cloudflare is perfectly content to forward the backbone traffic unencrypted over public internet. In fact, they advertise this malfeature.

This puts you into segment #3 in my list above. And I agree with you: the CloudFlare "flexible" SSL that sends last-leg traffic in the clear over the open Internet is a terrible idea.

But while that terrible idea opens up the potential for MITM attacks, it is not, itself, a MITM because the website owner has authorized CloudFlare to do that.

And of course there are levels of CloudFlare service that do properly re-encrypt the last leg to the origin.

> Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here.

You need to understand that you are not correct on the technicality. None of the protocols used in HTTPS are violated by decrypting or encrypting a session using the proper keys, even if it happens several times along the way.

Not every bad idea is a technical protocol violation. Not every bad idea is equivalent to an attack.

Abusing the term "MITM" makes it harder to talk with specificity about actual MITM risks. If CloudFlare itself is always a "MITM", then how do we explain why their Flexible SSL service is riskier than their Full/Strict SSL service? The former makes it easier for some mid-point to impersonate or alter the origin website. But now what do we call that? An "actual" MITM? A "real" MITM?

Let's just leave MITM to mean an attack. It's a lot clearer that way.

I think not having explicit support for proxies in HTTPs is a fundamental design mistake, so seeing how you’re keen on putting me in a camp, I’d like to volunteer for that one.

The term MITM is appropriate because it shows the whole thing is a charade. If handing off secret keys to third parties with absolutely no transparency past the first hop is the standard modus operandi then the entire protocol is a big fat joke. The only purpose, as far as I can tell, is to further centralize the web. First it was ICANN with domain registration, now it’s certificate authorities. Soon you’ll need a special permission from government to host a website.

HTTP(s) needs to be burned to the ground and replaced with something better. Liberty and freedom are at stake. How’s that for FUD.

Reply on news.ycombinator.com