We are under attack
en.greatfire.org
en.greatfire.org
Source: worked for AWS, was oncall during similar attacks. Nasty things with those they tend to start around 6-7PM (guess when does working day start in China).
The two colo centers we've hosted in have always helped us with DDOS issues free of charge. Maybe that's not normal, but even a former employee telling us to GTFO looks bad on Amazon to me.
In this case - 700k QPS (gigabits of ingress) of well engineered HTTP/HTTPS DDOS traffic is not something an average colo can or even will be willing to handle at all. I'm assuming a hot-potato DDOS, when a customer comes along with a long tail of colos and providers that already booted him. All that traffic, servers and ultra expensive engineer time. Everyone wants it for free, but ALAS.
Assuming you can find yourself a transit provider that supports BGP flowspec updates (many don't, sadly), you can do this fairly cheaply. You'd obviously want some level of support from a network tech that knew what they were doing, but it's not insurmountable. There's a bunch of other options available too.
This sort of thing is one of the downwsides of having your infrastructure managed by someone else. If things go wrong and your provider doesn't feel incentivised enough to help you out, there's a lot less you can do about it, other than just pay whatever sum they demand.
What I remember pushing BGP flowspec updates upstream was thought about as something close to impossible though.
My general point is: AWS is a business, and it operates as one. There are no hollywood style bad guys sitting there in cubicle dungeons on chests filled with gold thinking how to extract money, quite the contrary. It is understandable that customer cannot pay unlimited (from customers perspective) charges, but AWS pretty much incurs them, as customer being ddosed is consuming resources that would be otherwise be sold to others, or engineer time that would be put into developing new features and attracting new customers.
You can't disregard any business that doesn't fulfil that property as being "eventually unsustainable".
However my note was about general "traffic", if one sells video views for example, and revenues do not grow inline with adjusted to [almost always decreasing] bandwidth costs sooner or later that will become a big problem.
I suppose if you're not a Chinese hacker, it might pay to pretend you are by tailoring your working hours and days.
I know you can do this on the server, using many different techniques. But this does not help as the traffic still reaches you (that you have to pay for).
You can also do this with Geo DNS (and get much less of a bill).
And the ISPs, datacenters, and anyone with a router can block ASIA or China allocated IP ranges. Especially if it's not the type of a flood that's designed to attack the routers (instead of the web-server).
So what's stopping Amazon?
China is attacking them to prevent Chinese people from reading the website.
Your suggestion is to make the site unavailable to China.
Do you see why it is not a solution? You are basically setting up a market for censorship-- the attack doesnt ever have to end-- depending on how much China is willing to pay to keep the website offline.
Maybe I'm not understanding.
Attacking a single site is like more pro-active movement of the GFW itself, and a sign it can be so much worse.
Having provided IP transit at a largish network provider in a previous life, you have no idea at the complexity involved what you're asking for. It could be done, but the costs involved are non-trivial.
If you're honestly interested in the complexity involved, start reading about BGP, dynamic routing protocols, router/switch fabrics, control plane integration, autonomous systems, peering agreements, etc.
If ephemeral storage is a drive local to the virtual machine's host (which I think is the case) then having a cool down period would mean holding the hardware that used to be used by you in reserve until the grace period expired.
It's possible but it's a lot of work to solve a problem that can be better solved by not relying on ephemeral storage persisting.
Yes, it would mean holding the hardware in reserve for the cooldown period. I'm not talking months here, just enough time to recover from an accidental "sudo shutdown -h now" instead of "sudo shutdown -r now" (or similar). It'd be nice if Amazon sent an email warning about the condition and giving you an hour or so to go in and save your data/restart your instance. They could even make it a policy that you're charged for the time your instance is running + 1 hour to facilitate cooldown feature if they're really that worried about it; it's better than wiping data as soon as someone stops (from AWS console) or shuts down (from real console) an instance and providing absolutely no avenue for recovery, no matter how quickly you notice the mistake.
You have to specifically put something into the /mnt folder if you want it to be stored on the ephemeral storage. Any other location is safe and will persist through halts and stops.
In practice the only thing you should ever use the /mnt folder is maybe a Nginx disk cache, or as an alternative /tmp or something like that. Basically if stuff you don't want to lose is finding its way onto the ephemeral storage then you are doing something wrong.
There are plenty of plausible situations where an AWS user can find themselves with important, even just temporarily important, data on ephemeral. Whether those are the result of "correct" usage or not, it's beyond the pale to just zap that data away and tell the customer tough titties as soon as a shutdown command is issued.
While you may not like the trend, and some providers have better options than others, and you may require some operations over others... but very few businesses can afford to manage multi-site infrastructure that can dynamically scale. Most are probably served just fine with rented servers, traditional colocation, or don't even need more than one VPS...
That doesn't make the technology bad, and only makes you seem ignorant in your prior statement.
I'll go into it later, but for the most part I consider the cloud a really bad idea. I also regard most cloud companies as "buzzword-enabled" so as to attract investors.
My gripe with scrum and agile is not so much with the methodologies, but with companies that think they have a methodology when in reality they have a bureaucracy.
You mean the company that provides caching services for pay DDOS services?
I recall only hearing about one time when packets from Chinese ISPs were completely dropped for some reason and only for short period.
I've also have an anecdotal reference that one can persuade providers that actually deliver traffic from China to do filtering on ingress on next hop routers after China, but that should be something very serious, that impacts their revenue as well and prolonged. As another commenter noted - costs for providers are very non-trivial.
In my experience DDOS is always money competition, it costs money to mount one, it costs money to defend against one. Unfortunately when one of the sides is [allegedly] a country it doesn't play very well.
I was working on an anti DDoS system for SIP, a UDP-based protocol. Basically the options were: 1. lockdown, just whitelist known good customers, and break many scenarios. 2. Attempt some kind of analysis, like sending out probes to determine good/bad IPs. 3. Scale the hell up. Write L7 stuff that can go at wire speed, and get lots of wires.
Needless to say, #1 is the easiest to implement, but allows you to get your pipe saturated. #2 requires compute + pipe, and #3 is the only thing that'll really work.
This matters because DDoS'ing a telecom can be very lucrative. I can say with good confidence that demonstrating DDoS capabilities are probably worth 5-6 digits in blackmail against many companies.
If they put a reCAPTCHA wall in front, the GFW can simply block reCAPTCHA (easy -- it is a Google property and they block everything else from Google anyway) and no one from China can access Greatfire without a VPN. Mission accomplished.
If the goal was only to block Greatfire for non-VPN users, then they could just use the GFW for that from the start. The use of a DDoS can only imply that China wants the site offline for everyone, even VPN users.
With this DDoS, they are taking the different route of attacking the infrastructure of Greatfire such that they can't serve traffic from China at all. Causing massive bills and outages for Greatfire is probably a bonus, but I don't think that is their main intention.
Alternatively, call CloudFlare.
Don't just absorb this through Amazon.
"CloudFlare, which offers content-delivery network services, said last week it cut off Lantern’s use of the service, saying it was unauthorized. “We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
http://www.wsj.com/articles/u-s-cloud-providers-face-backlas...
I'm not very impressed. Maybe someone from CloudFlare is around to defend that position further.
There's a popular idea that businesses (and people) have no responsibilities to anyone but themselves, because what they have is theirs; they built it themselves. But if you think about it a little, it's obviously false. Here's a more accurate statement:
We're a tech company whose success is completely dependent on the freedoms in our nation and many other nations around the world, and on the political and economic systems, infrastructure, and enormous wealth that blossomed from them. Without the sacrifices of blood and treasure by our predecessors of hundreds of years, and of many people today, we would not have these resources or opportunities today. There are many talented people born in many countries who, without these benefits, have no opportunity for success.
They can't sacrifice their company for every principle, every time, but there's a middle ground between that and 'we're just a tech company so we have no responsibilities'.
It's not just a popular idea, it's why they are created as firms instead of philanthropies. There is a difference and it does matter what the expectations of the donors/investors are.
> We're a tech company whose success is completely dependent on the freedoms in our nation...
This sounds great but how is it reflected in company policies?
> They can't sacrifice their company for every principle, every time, but there's a middle ground between that 'we're just a tech company so we have no responsibilities'.
A company could easily make a statement to its investors about its moral stance on issues that it expects might harm the bottom line.
The company does have responsibility to its investors not to go rogue and burn cash just because it feels good. Most of the time the kind of corporate behavior that you praise is actually clever PR that costs the companies little.
It may not be coded into law, but it is still a true statement.
A false dichotomy.
I'm curious about your reasoning behind this statement.
In other words -- on the blackhat-whitehat scale, it's either black- (or at least very charcoal-y grey-), or whitehat. But I just don't see modern, large companies generally acting that way -- not because they're led by altruists (they're certainly not); but because that's just not human nature (across the board). Most of us are greyhats (somewhere on the scale); and the behavior most business leadership I've either read about, or seen directly (behind closed doors) seems to fall somewhere on the greyhat scale, also.
That is: large business definitely aren't philanthropies -- but in general, most of them (even many of the traditional "bad boy" players like banks, big pharma, etc) -- aren't straight-up moral nihilists, either.
At least that's the way I observe these things. I could be wrong.
Businesses should act within the law, and lawmakers and the public determine what legal safeguards are necessary. For example, if you start a restaurant you must comply with health code, fire code, etc. If you start a bank you need to keep a certain amount of risk capital, etc., etc.
One could argue that all dishes used by a restaurant should go through hospital level sterilization, or that banks should contain more risk capital than they are required to by law. Such arguments would be in the name of safety or quality.
One could similarly argue that restaurants should use at least 20% locally grown produce or that banks should lend 20% of capital to underprivileged groups. Such arguments are in the name of moral responsibility, etc., and lawmakers have actually implemented many such laws for banks.
For an investor who wishes to invest in a bank or a restaurant, there are many options. Being able to compare financials and other metrics will help the investor figure out which is the smartest investment (based on her risk appetite, etc., etc.)
Why might a restaurant decide to focus on locally grown produce or a bank decide to focus on its ethical treatment of subprime borrowers? Largely for PR/marketing reasons. If such marketing campaigns are successful, customers will flock to the bank or restaurant in question and (assuming they are still able to be profitable) make the bank or restaurant a more desirable investment.
One can pick any business and any metric that he thinks has moral significance and claim either "regulators should require x, y, or z" or that "that practice is horrible". One might be right... essentially ahead of the game morally from society's average.
The perception of moral progressiveness, like the font chosen for a brand, is one factor that helps determine a business's success. It may be the case that most of the meat we eat was raised in unconscionable conditions, or that 30% of imported electronics were assembled by modern serfs in near-slavery. The more we are aware of such things, the more likely firms are to make the most progressive choices.
Cloudflare, I'm sure, will happily ignore any laws like that. The question is: why not ignore this too?
Jurisdiction.
The degree to which they can practically enforce that jurisdiction becomes a game of relative power and how willing others are to constrain it, of course.
EDIT: It turns out Lantern was using an exploit at Cloudflare [+], and wasn't a customer. My apologies /u/eastdakota.
Given that the country you live in has violated human rights to some extent, and that you could reduce your contribution to that by not earning taxable income or purchasing taxable goods, is it not also your defacto position that you value revenue over human rights?
(My apologies in advance if you've ceased paying taxes, or buying anything taxable, or if somehow no one in the world believes your country has violated human rights, or might do so in the future, or your position is that revenue > human rights)
My point is that the world is a lot more grey than you make it out to be, and that you are also in some way likely valuing revenue over some human rights abuses.
That was then, though. Now, when skilled labor is of greater importance and unskilled labor of comparatively little value, I submit that things may well be very different.
Not commenting on what CloudFare should or should not do, just indicating that your high horse actually has longer legs than you gave it credit for.
My comment is about that concept in general, not about Cloudflare in particular. I don't pretend to know and won't judge Cloudflare based on one sentence taken out of context. For all I know they are excellent members of the community; in fact they could do be doing good things behind the scenes without publicizing it, which might be wise if they are as exposed to China as other commenters say.
Again, I realize this place isn't exactly a bastion for this kind of sentiment, but have some thought for freedom here, CloudFlare. The US may suck at helping a lot of the time, but if you've got a group of folks trying to deliver some good ol' freedom to a country like this, and that country is trying to shut them up, maybe put out a helping hand, or at least don't shut off service.
Come on...
In the case of Lantern, they were taking advantage of a bug in our system. Specifically, they were setting the SNI field (outside the encrypted packet) of a request to look like it was going to an actual CloudFlare customer (e.g., news.ycombinator.com) and then setting the host header inside the encrypted request to point to some restricted site. The bug was that we did not check that the SNI field matched the host header, which allowed Lantern to do what they were doing.
Lantern was not a customer of ours, instead they were exploiting this bug to essentially disguise traffic to look as if it was coming from one of our actual customers. One of our biggest concerns was that this would put CloudFlare's actual customers at risk of being blocked. And, beyond that, even if it weren't being used to avoid Internet restrictions, that someone could effectively impersonate the identity of a customer on our network is, per se, a flaw that we should patch. As soon as we became aware of the issue, we began matching the SNI header to the host header and, effectively, patched the bug.
We've always been very supportive of a free and open Internet. However, even if we support what someone is doing, we can't put our current customers at risk of collateral damage or keep open bugs that allow our network to be exploited.
Matthew Prince Co-founder & CEO, CloudFlare @eastdakota
This makes a world of difference.
Just to confirm, does this mean that if the exact same attack had happened, but Lantern had been a CloudFlare customer, you wouldn't have shut them down?
Still curious about this quote: “We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
So if Lantern were a customer, would the outcome still have been the same?
Usually patriotism is the last justification used by those who have nothing else to stand on, like the KKK trying to oppress African-Americans, or the Nativists trying to oppress Irish immigrants, or modern-day politicians who decry all Islamists as terrorists, or the border states trying to oppress migrant workers, etc. Each time they've exhausted all other excuses, Patriotism is the last justification for their actions. (I won't touch on Mao, Stalin, Hitler, etc because they're too tied to specific nationalist policies)
Personally, I wouldn't want to identify myself as a Patriot, because usually they're the ones standing on the wrong side of history.
Unless you were just trolling.... ;-)
Actually, it is. Patriotism, in being a Patriot, is a loaded word in the American (USA) context. Specifically, it is about doing what is good/right for the country and her citizens regardless of the law (i.e. British rule.) Or so says my recollection of American History. I mean... just look at the Patriots (rebels, in the british colloquialism) in the image on the wikipedia page for Patriot_(American_Revolution).
"The Oxford English Dictionary third definition of "Patriot" is "A person actively opposing enemy forces occupying his or her country; a member of a resistance movement, a freedom fighter."[1]. In this definition, if the alleged DDoSers are Chines, attempting to block the actions of a foreigner imposing influence in their own land, they are the more Patriotic? Which is why the term is utterly useless in this argument; Dare, any other.
> Usually patriotism is the last justification used by those who have nothing else to stand on[sic]
Thus was it written.
[1]http://en.wikipedia.org/wiki/Patriot_(American_Revolution)
edit: add ambiguous ?
> Due to the sensitive nature of the content on our web sites we prefer to remain anonymous at this point
If they want help they need to be transparent about who they are and what their objective is. One man's tool of diplomacy is anothet man's... etc.
I might have to reconsider mine and my clients choice of providers for this very purpose.
[1]: https://blog.cloudflare.com/one-more-thing-keyless-ssl-and-c...
Same thing with anyone who operates in China. The only difference is China is more transparent with their demands.
Response from CloudFlare's CEO here: https://news.ycombinator.com/item?id=9234367
"Often these attacks appear politically motivated — going after, for instance, citizen journalists reporting on government corruption. The promise of the Internet is that it is a great leveler — that anyone with an idea can reach a global audience. These attacks threaten that promise."
[1] https://blog.cloudflare.com/protecting-free-expression-onlin...
That's really rich, considering CloudFlare happily takes money from booter services. These guys are scum, I have no idea why HN fawns over them.
I'm guessing in this case it's simply a case of them choosing which battles to fight. They probably don't want to commit to run an open proxy for everyone in China to access banned websites. That would likely get them banned outright in China, which, for a CDN like Cloudflare, would really hurt their core business.
Try Incapsula, GigENet, Blacklotus, Cloudflare first.
They're the most abhorrent combination of incompetence and arrogance that I've ever met in the tech industry.
They're the Oracle of the network world. Just don't waste your time on them. There's plenty better and cheaper CDNs nowadays.
Leave Akamai to the Governments and MegaCorps, they deserve each other.
We've tried a number of DDoS filtering services, and Proxleic/Akamai has been the most professional and effective so far. They're also the most expensive, by a fair bit, but you get what you pay for I guess.
https://www.hetzner.de/gb/hosting/produkte_rootserver/ex60
Such a single server, isn't the same as a CDN of course, just curious what you consider "buckets of cash". I see cachefly asks for 400 USD for 2/TB a month, for comparison.
First thing I'd have done is take the site offline and call the various DDoS mitigation services (Incapsula, Cloudflare, etc.).
Pretty surely most of them would gladly pick up the slack here, given the free PR (possibly even in mainstream press) they get in return.
(full disclosure: I do devops for one of the Russian sites that is currently mirrored by Collateral Freedom)
And pretty certain that there's zero PR value in those stories.
As for PR, there absolutely is good will towards organizations that take on censorship. This is the #1 story on HN, so a company stepping in and saying "we got this covered, free of charge" not only shows they are good people, but also gives would be customers an idea of the quick implementation cycle for their CDN + the load it can handle.
Real world use case + helping to stop censorship = great PR
Now in a way their $ penalty with this attack is not unlike growth. To me it sounds like they invested in their branding and will have a name as a company that can sustain damage and keep its ground - important because their goal is to be credible in bringing content in/out China.
What if they had given in? They'd be yet another victim of an attack sort of linked with China.
I do see a business sense behind what I assumed was their moral stance and I definitely stand by it.
Unless your attitude is worth $20 USD per minute to you then yes, that's very much what you want to do.
And pretty certain that there's zero PR value in those stories.
Well, Greatfire seems to be betting a 6 figure stake on the opposite here...
https://www.ovh.com/us/dedicated-servers/enterprise/2014-MG-...
Thanks for posting :-)
I've been looking for provider possibilities for my next failed startup. I'm not sure how they can deliver for that price but who am I to complain!
http://www.investquebec.com/quebec/en/financial-products/smb...
Basically the Quebec government pays 24% of your company's wages, up to $83,333 per employee!
And if you don't need any support or server hardware, give kimsufi a look.
I do not work for OVH, but am so impressed by their automation and value in this field I refuse to use anyone else.
http://www.webhostingtalk.com/forumdisplay.php?f=2 (and most of the rest of the site).
http://www.ovh.com/ca/en/a1171.protection-anti-ddos-service-...
I moved from iweb because they null-routed my server during a DDoS attack that happened for no apparent reason. (My guess is that it was a user upset at some user-generated content, so they decided to DDoS my server rather than notifying me).
Takes about 5 mins to kick in once an attack has started, which is very good.
――――――
¹ — http://www.wolframalpha.com/input/?i=1%20GiB%20to%20MiB%20an...
Disk space even obeys this now even though it was only done by marketing so they could reduce the amount bits delivered while charging the same.
I can't understand how people can be so delusional to think that randomly redefining prefixes is a good idea. It was never, ever used consistently, and JEDEC just should get rid of this idiocy.
This is why there is a need for the two different prefixes. If I am selling a 256GiB SSD and you are selling a 256GB SSD, those are now differentiable on their face.
Standards bodies aren't anything magical, and I don't get the slavish following they seem to get. So an RFC says something, or another group mandates something. BFD. Unless you're expecting interop to work, use standards as you see fit. They aren't an ends unto themselves.
In this case "GB", when referring to RAM is unambiguous. Only disingenuous cloud providers or petty editors would use a base 10 interpretation.
(1gbps ~ 300 TB, 100mbit ~30TB/month)
Oh really? :-)
Perhaps the US State Department might be inclined to help?
Take a look here: https://en.greatfire.org/news.ycombinator.com
That would certainly be my guess.
Step 1 : Unleash DDos Attack against target
Step 2 : Wait for the the target to become overwhelmed and ask for public assistance
Step 3 : Contact the target under the guise of being able to help
Step 4 : Win trust of Target after "mitigating" the attack you are actually in control of
Step 5 : Repeat until enough access has been gained
Hopefully making the front page will at least get them the attention of Amazon or enough donations to cover the temporarily (absurdly) high operating cost.
Edit: I don't actually know if this is still true and on what scale, I just know that's it's true for a website I use according to chinese users.
The bigger problem is their stance on Latern mentioned above.
Maybe the corporations might be more stable and have more integrity, but you have to realize that China is the largest economy in the world now, and they can flick Amazon out of their market with their pinky and not even blink.
http://www.linuxjournal.com/content/back-dead-simple-bash-co...
Should a group of people in democratic, Western countries be able to subvert the will of a world superpower with impunity?
Of the two scary worlds, I guess I'd rather choose the latter. But I don't even like having to choose.
(I doubt Amazon like being asked to choose even less, and I would be surprised if they cut you any slack. Sedition is not looked upon favorably, and abetting those perpetrating it is not either.)
Crazy time we live it that this is even possible.
The ability to increase freedom in a foreign state is related to the ability to prevent a decrease of freedom in your own.
Inconvenient websites help uphold the duties of the fourth estate when mainstream media outlets have seemingly abandoned--or at least heavily de-prioritized--those duties. It is important for all governments, not just those with sketchy human rights records, to know that even the mightiest machine can be taken offline by a single wooden sandal.
Deleted comment
Edit: never mind, figured it's obvious.
Aren't DDoS requests pretty much simple GET requests? Is it not possible to determine which requests to serve and which ones to ignore?
https://blog.cloudflare.com/understanding-and-mitigating-ntp...
If the attackers are just sending GETs you might be able to filter them out, but if they're sending random packets you usually need upstream help to keep them from getting to you to begin with.
There are also things like Slowloris that just use up resources vs using bandwidth, those are harder to identify but easier to deal with on a server-by-server basis.
There are ways to mitigate this but it requires being able to analyse current traffic and past traffic quickly, and at scale while having the expertise to set up firewalls and other filtering correctly
Not to mention when the people who don't like you have the resources of nation states.
http://gizmodo.com/china-finally-admits-it-has-an-army-of-ha...
anyone remember that story a few days ago where China said "Hey, we want NSA-level backdoors in hardware, too!"? can someone help me with a link?
wake me when the clickbate is over.
- Use memcached+nginx to load the drupal content instead of calling up PHP each request (PHP saves the page in memcached, nginx reads it from there).
- put it all behind CloudFlare
This works for Wordpress too.
http://en.wikipedia.org/wiki/Coral_Content_Distribution_Netw...
It's still possible to overload that capacity, but it's a lot more than what any standalone web server can take. Furthermore, the web server itself will stay online, as it isn't actually getting hit by the flood of requests.