When “Dumb Pipes” Get Too Smart
saurik.com
saurik.com
Given the volume of traffic HN sees, and given their desire to not be DDoS'ed, it seems to be a necessary step, that is not over-rated.
To be clear, you'd see similar results with any CDN sitting in front of HN... The benefits really start to show (from an end-user's perspective) when you're on poor connections or are far away from their (HN's, in this case) server/cluster. HN sees immediate benefits for the things above, plus it saves them a lot of bandwidth.
Instead of having a mechanism by which a website under attack (or simply un-monitored heavy load) from a host or number of hosts can direct the ISPs of those hosts to not send it traffic the CDNs instead use the present lopsided nature of peering agreements to simply sink the hits.
In the above mentioned solution either the direct ISP would filter out the requests before they hit a higher tier ISP/backbone, or an ISP that does not provide said filtering would (possibly blacklisting the entire client ISP for sufficient bad behavior).
How is this any different? You're putting the responsibility of being internet police on the ISP's, which have shown their either do not want that responsibility, or will abuse that responsibility.
Not to mention, this would go squarely against the notion of ISP's as "dumb pipes".
abuse@ ought to function and produce a rapid response.
(That doesn't mean we shouldn't have anonymous, untrackable services as well, but using such services means you have to put up with spam too. When you have a service like email, with all the tracking information readily available in the headers, it shouldn't be as near-impossible as it currently is to get something done with that information.)
The first is large networks that tolerate spam, and those are irrelevant because their IP blocks are already on every blacklist in the world.
The second is compromised machines, which are the real problem because they're ephemeral. Spammers can compromise a hundred new machines a week. You can't fix them or block them faster than they compromise new ones. The only solution to that is to improve computer security so they don't get compromised to begin with, which has nothing to do with ISPs.
Removing people from the internet is never the answer, both because it's too broad (should the spammers not be able to go to the government's website to pay their taxes?), and because the nominal current source of the spam isn't actually where the spammer connects to the internet anyway.
The better solution if you can actually find a real life spammer is to impose a fine on them that exceeds the profits of spamming.
Really, if your computer is spamming someone, even if you aren't aware of it, you are harming them, and ignorance shouldn't be protection. Maybe fines and jail are a bit too severe, but rate limiting and possible disconnection are more then fair.
It's a pretty solution to the problem in theory, but when you actually apply it, it doesn't quite work out as well.
(2) Same with spam blacklists, you get de-listed either automatically, if few weeks pass without incident, or faster by manual request.
(3) It should not matter why bad traffic was sent. A much more realistic situation is: "So you work from home and your computer gets trojan which starts to send phishing emails, so now the ISP cuts you off from the Internet. Now what?"
In either case the answer is: prove to humans @ ISP that you are no longer danger to the internet and ask them nicely to unblock you. If you work from home is that critical, have a separate internet source (say 3G modem which you activate with a phone call)
They can literally compromise new machines faster than you can identify existing compromised machines. It doesn't do anything to impose anything on the machine owner -- as soon as you identify the machine you can blacklist it, and as soon as you blacklist it they stop using that one.
The problem is in the time it takes you to identify a compromised machine, they can compromise many more additional machines.
That's exactly what I'd like to see. But that does depend on cooperative ISPs.
How does that help? The ISP can tell you where the compromised machine is, but it will be a different compromised machine in an hour so that's no help. It can't tell you where the actual spammer is because they're behind five proxies in six countries.
Often it's more feasible to apply costs to a higher level of responsbility than the party directly responsible. E.g., sanctioning a country which, say, harbours pirates or highly-polluting industries.
It's the model of collective action and pressure.
Netflow data is extremely helpful, although not on its own (it will also identify legitimate customers running their own mail servers). It's a good start, though.
There might be other ways of managing traffic, though it would require more pipe smarts.
Generally, I'm in favour of ASNs -- autonomous systems -- taking responsibility as you describe. After all, they do present a single administrative bound of control, and ought take responsibility for their traffic as you describe.
The Routeviews Project (asn.routeviews.org) offers both DNS lookups and downloadable zonefiles for mapping individual IPs to ASNs.
Of late, the problem I'm finding is that far too many services are hosted on Amazon, though firewalling off a wide swath of a few AWS AZs might be an interesting approach.
In the real world making large volumes of things appear at one specific location takes effort. The real world is also fairly good about having ways to track down and prosecute those who are unusually heinous about some activity. There are costs, resources, and attachments of presence involved.
In this context the closest example would be the PTSN (Public Telephone Switched Network). That analogy still fails because while the cost is low there is still SOME cost and a lot of documentation about who might be 'making someone's phone blow up'. However from the perspective of the solution it still holds true.
On many hard-phone-lines it is possible to dial a number sequence to block calls from the number which previously called the phone. In this case the provider hired by the victim will deny the connection request before it ever reaches the victim.
Something similar can currently happen as 'blackholing' the victim, but this is victim blaming and allows the attacker to win. What is necessary is to instead push the blocking far enough back the chain that the victim sees no bill and is completely impugned from the actions of bad actors. Logically, this means blocking it at the backbone level or lower, and still charging the sending party for the data.
If you tried the same thing with an image-heavy site, you would probably see the opposite result.
Instead, CloudFlare relies on crazy content transforms that started as essentially "what if we run mod_pagespeed for you in the cloud as a service" and then expanded in scope from there. I mean, even when they cache things, as I understand it their cache hit ratio is extremely poor (whether due to restrictive cache sizes or limited retention windows, I don't even venture to guess).
This means that if your content isn't inherently slow (using poorly compressed images and tons of fragmented JavaScript and CSS files with broken ETags being consumed by HTML with script tags in "the wrong places", none of which is minified), CloudFlare is going to make your site slower.
Really: the two things they ended up getting stuck in the ecosystem for were 1) being free or at a fixed low price for supposedly infinite bandwidth (though people report that when you actually use a ton, CloudFlare claims you are being attacked and need to upgrade to their protection racket, which leads us to), and 2) performing DoS attack protection by using a ton of annoying heuristics and IP filters and JavaScript delays and even captchas that you should notice no "real" site behind a normal CDN ever seems to need, and yet somehow is a hallmark of the experience of "this site is using CloudFlare".
One of the challenges here is popularity: if you're a major site, yes, Akamai will have content cached at most of those 140k POPs but sites with different blends of cacheability and global traffic distribution might see a better cache hit ratio with fewer POPs so things stay hot in the cache longer.
These are complicated services so the results are going to bay widely depending on what you need (e.g. having an optimized site means that every optimizer service I've tried made performance worse), and how much you can adjust your application to work with the CDN and tech changes like HTTP/2. The other reason why blanket statements aren't worth the time needed to make them is that these are not free services and you can get easily find cases where service A is better than service B but not enough so to justify the price.
(Add for the DoS tools, note that that's pronounced “feature” by most people and at least personally I never see them except when testing through a commercial hosting company's network. Tor users whine a lot about them but that's just entitlement and unwillingness to think critically about what a site owner sees from their peers.)
As for their DoS "features", yes: they have a bunch of features because it makes them look like they are doing something worth value, but I am going to again remind you that somehow you never see websites that are using normal CDNs suddenly give up and give you an Akamai-branded captcha, and yet somehow they do not crumble under the weight of denial of service attacks; it just isn't the case that using CloudFlare is the only way to be safe from DDoS, and there's something weird about how the experience of using a CloudFlare website somehow is noticable.
A fair comparison would be hacker news with vs hacker news without cloudfare?
A CDN is extremely valuable for a website that has cacheable content, since it allows a site to scale dynamically with increased load. This is the same reason people use AWS or any cloud provider; a website has to have the capacity to serve their highest peak traffic, but it is needlessly expensive to maintain that much infrastructure around the clock.
Yeah absolutely this. They spin everything they do as some kind of heroic "for the people!" decision even when it's just about cutting costs or not having to solve "hard" problems. One example are DNS "any" queries. Cloudflare just decided to toss standards out because they aren't up to conforming to them. As far as I'm concerned, this Cloudbleed thing is karma, and nobody should believe anything Cloudflare says about itself.
What I'm thinking they could've done in the case of ANY is to respond with truncated and switch to TCP mode at which point it becomes harder to do the spoofing dance and (ab)use ANY as a DNS amplification attack to DDoS a target. Unfortunately that does put an extra cost on the DNS server which might also be undesirable. However that would've probably been a worthwhile tradeoff.
There's an IETF draft that attempts to provide some guidance in the area of the ANY query: https://tools.ietf.org/html/draft-ietf-dnsop-refuse-any-04
> Well, two out of the three people proposing that document work for Cloudflare. The other works for Dyn.
Two companies that have some experience with dealing with DNS attacks. Their motivation may be self-serving, but they're not the ones doing the attacking.
WebCore is doing what it's told, just just being told very stupid things (very, very quickly)
No, the bug is definitely in the browser. Web code is untrusted and should not be able to adversely affect the browser.