Pastie.org host pulls hosting after DDoS attack
pastie.org
pastie.org
Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. Your idealism will fall down quickly when one annoying customer threatens service for every last one of your clients: it becomes a "do I continue to get paid or do I fight for this customer" equation.
Price out mitigation equipment and the multiple high-level engineers you will need to administer it before you respond telling me how wrong I am, by the way. The gear alone is an engineer's salary just to get started.
Edit: Fine, I dropped the Amazon example.
http://arstechnica.com/security/news/2010/12/wikileaks-kicke...
> The Wikileaks website migrated to Amazon's cloud hosting service yesterday after being hit by a distributed denial of service (DDoS) attack. Amazon decided to discontinue serving the controversial website this morning in response to pressure from critics, including prominent members of Congress. ... Senator Joe Lieberman (I-CT), chairman of the Homeland Security and Governmental Affairs Committee, was among the congressmen who pressured Amazon to stop hosting Wikileaks. He told AFP this morning that he plans to question Amazon about its relationship with Wikileaks.
I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.
Granted, not everyone knows how to do that in a pinch, but it shouldn't be a big deal for a hosting company. You only have to figure out how to do it once and when you detect a DDoS attack, flip the switch for that customer.
That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle.
As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time.
In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry.
I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.
If you're a "typical startup" with an Amazon footprint, you have no mitigation strategy for flooding attacks aside from not attracting them. If someone points multiple gigabit at you, there is just about nothing you can do except hope Amazon can do something.
There's a range[2] of ISPs who will sell DDoS protection to you, either as an addon when you host with them, or as an external service (re-routing your traffic).
E.g. StormOnDemand just recently added it to their portfolio[1], which is note-worthy because they actually list prices right on the website.
Either way, even without "explicit protection" any ISP beyond mom&pop-size deals with these attacks every day and will sort them out for you for free the first couple times. Only when they turn into a habit or become so huge that they have to talk to their upstream they will politely ask you to throw some money their way.
Pulling the plug immediately is definitely not normal. However considering Pastie was apparently a sponsored account it's at least somewhat understandable (albeit a terrible PR move).
[1] https://www.stormondemand.com/ddos.html
[2] When in doubt, and pockets deep enough, there's always Akamai. They're the ones who can filter TBit/s-scale (yes, that was a T) attacks for you.
Nulling immediately is. You're also assuming that this is Pastie's first DoS attack, which we don't know based on the information presented to us.
For serious accounts (in the 6 digits/year) absolutely not, unless the attack is large enough to affect other customers.
Admittedly RailsMachine looks very small, in all likelihood their pipe was rather easily clogged and they simply didn't have the choices that larger ISPs have.
If it doesn't affect other customers, a hosting company won't act or even be aware, in most cases. They'll just send you a bill for the transfer. If someone attacks you and it impacts other customers, you get nulled. I'm aware of 7 digits/year and 8 digits/year accounts through industry anecdotes that have had machines nulled. The engineer operating the null doesn't say, "oh, that's X, maybe I shouldn't fix the network for my other customers".
I don't understand what you're disagreeing with.
There's a bit of middle ground between "sending a bill" and nulling.
I've been hit by two larger attacks in the past (GBit/s range) and the respective ISPs were both extremely supportive, switching our IPs while they tightened their filters. Neither billed us a dime despite our ingress spike making quite a bump in their charts and a lot of handholding over 2-3 days.
In any case, thank you for a very informative post.
https://www.gigenet.com/order/index.php?order=true&form=...
95% of dedicated hosts out there will immediately nullroute you if you get a DDoS attack like Softlayer, Ubiquity, etc... Unfortunate as it is, this is a common practice for hosting companies.
At the risk of getting downvoted again, I am going to re-paste a comment I made.
DDoS attacks are way too easy to do now, and something needs to be done about it.
Anyone can go to www.hackforums.net for a free UDP flooder (called shell booters there), or rent one capable of over 20 gb/s for $5.
Or if you want to do it yourself, pick a cheap throwaway vps from www.lowendbox.com, go to www.gametracker.com and grab IPs from COD4, Wolfeinstein ET, Medal of Honor, etc... and send UDP query packets with your IP spoofed as the victim. This will amplify the size of your attack 20x or more and hide your IP address. You can easily get 10-20 gb/sec attacks like this.
A little off topic, but I've always felt a slightly uneasy about the concept of "attracting" DDoS attacks. Sure, if you knowingly piss off a bunch of script kiddies, you're attracting attacks. But it seems that nowadays, any site that hosts user-generated content is at risk of being attacked for any random reason. And yet, a lot of people talk about "customers who attract attacks" as if those customers are to blame. It almost sounds like blaming women who wear certain types of clothes for attracting sex crime.
Of course, the fact that you didn't do anything to provoke an attack might be irrelevant when your upstream faces a choice between cutting you loose and eating hundreds of thousands of dollars. People need to do what they need to do to protect their networks. Nonetheless, I'm curious what you guys think about the concept of "attracting attacks". If you blame the victim even a little bit, does that affect your judgment about what should be done in the case of an attack?
I completely stopped reading this comment here, when you wrote this, because that was a completely off-base comparison and has absolutely nothing to do with the topic at hand. Worse, you probably know it; I wouldn't assume you to be stupid. And that was a mountainously stupid comment.
In hosting, there are customers that attract DoS attacks. Period. Ask anybody who does hosting. IRC servers are a canonical example and are DoS magnets. Torrent trackers are another. Pastebin sites, like Pastie, are becoming another (look who uses pastebin.com a lot: Anonymous). Hell, Facebook and Google probably takes several DoS attacks a day just by nature of being well-known.
Don't even conflate my argument with an undertone of sexism.
But I don't think your unwarranted indignation adds anything to the question that I was trying to ask to other HNers. Unfortunately, that question was at the end of my comment, past the point where you stopped reading.
It is as ridiculous as banning airports because it attracts suicide bombers.
The longer hosting companies put off developing a real solution to this problem, the more DDoS attacks are going to happen because they are so effective at getting servers kicked.
I admire the people willing to fight the good fight, but it doesn't seem like the guy running pastie.org has any skin in the game. It's easy to decide you're going to run a laissez faire type site when you don't pay the bills.
Even expensive and large hosts like Softlayer will null route you.
They just don't want to pay bandwidth or go through the hassle of asking their upstream providers to filter the attack for them.
Also, why did Rails Machine throw out the site so quickly? If I choose to sponsor someone out of my free will, I'd do so without distinction from paying customers.
The attack might have been launched for the fun of it, too. Why not, when you won't be affected personally?
On the other hand, it's also possible that the originators of the DDoS are angry about something the hoster did take down. Perhaps they'd put up material against the rules, Pastie took it down, and in a tantrum after-the-fact, they decided to retaliate with a "well if you won't host our stuff we'll block eveyone". (This is a bit more like the Anonymous DDoS against payment services they didn't like.)
Without a statement from those involved, it can be hard to determine DDoSer motivations, but those are some common patterns.
>"why did Rails Machine throw out the site so quickly?" If you run a datacenter, you pay for an uplink. That uplink has limited capacity. 4gbit, 10gbit...whatever. A big attack can saturate that link completely, so even with the biggest most expensive "mitigation device" on the market (some of this gear can get into the hundreds-of-thousands-of-dollars for /one/ device, mind you), if a DDoS is overloading your upstream bandwidth providers, you can either have your entire DC brought to a crawl, or null route the site.
With that said, how did CloudFlare keep lulzsec up? Anycast, lots of iron, lots of smart technicians, and probably tens to hundreds of thousands of dollars in bandwidth fees. TL;DR it was a publicity stunt that they very smartly played up.
DDoS is pretty misunderstood, and lots of clients think that there is some magical box that can take all the traffic. Again, if your link is saturated, a "mitigation" device can only filter the traffic; your upstream providers can and will take you offline if you don't fix it. Failing that, you get a massive overage bill and every other client at the facility is crawling. It's not really a good solution (mitigation devices /can/ help with smaller attacks, but for the big stuff, null routing is the best solution unless you have something like CloudFlare -- and even they will pull the plug if the attack gets too heavy, because it's simply not worth the expense to them to keep your site online.)
Edit: Let's say you run Joe's Web Hosting. Joe's has three facilities, and you run redundant ten gigabit uplinks at each. Last I priced a device that could handle ten gigabit at line rate, it was ~$120,000, so figure:
$120,000 x 2 uplinks x 3 facilities = $720,000
(I honestly don't remember if that figure was for the gigabit device or the ten gigabit device. I think the ten.)
network equipment vendors like to have high list prices in order to:
- ensure they get sales inquiries from serious buyers;
- give sales representatives a large amount of leverage when negotiating the actual line-item pricing for each purchase order;
- promote some kind of enhanced "value" to executives ("it costs a lot of money, so it must be good!")
a realistic price for two routers with two 10gig-e linecards is about $15,000-$25,000 per unit direct from brocade. cisco would be around the $35,000-$50,000 per unit ballpark. if you're paying any more than that, you're getting extremely shafted.
further, the circuits for the uplinks aren't going to cost that much themselves, as it's all sold under percentile-billing (not even 95%, usually more like 90% or 80% at over-gig-e levels these days) when we're talking actual transit costs. commits for "Joe's Web Hosting" might only be 1.5gig on each 10gig circuit, so that cuts down pricing a lot right there.
in terms of getting an experienced engineer, most attacks can be detected and mitigated automatically using netflow/sflow/ipfix flow analysis. i wrote software which handles the detection automatically, with a fairly high success rate, and it's free to use on bitbucket. :)
I worked for a serious buyer when I had this casual conversation with the vendor. You do business with my former employer (I've spoken to you before, when you took over cia.vc). I'd consider those facilities "real" datacenters, and they're certainly the same size, if not significantly larger, than your hosting outfit. No need to appeal to authority with me, honestly, I think we're on the same page -- I think your comment misread me as pricing routers instead of mitigation gear.
As for flows, I was interested in a solution that I didn't have to write software to implement. I'd rather have a supported appliance that can handle figuring out DoS attacks itself, rather than me parsing flows and feeding that information back. That way, if the software doesn't work, I can blame somebody else rather than me. My time is precious. Yours sounds less so, and that's your prerogative.
in terms of mitigation hardware, all of those appliances are a ripoff. the solution is to replace them with free software that does automatic analysis on the flows and then sends that data automatically elsewhere, putting them out of business.
i have a system set up that does automatic mitigation using ddosmon by having a "scrubbing center" running on freebsd-based devices. this is accomplished by having a custom 'action' module in ddosmon which does three things:
- calculate the necessary ACLs
- insert them into the appropriate pf tables so the mitigation strategy fits my ruleset
- direct the router to send traffic for the IP being flooded to the scrubbing appliance
this is basically the way that the mitigation appliance vendors tell you to do it if you're handling >10gig floods anyway.
i think ddos mitigation is really a place where free software can cause a massively needed paradigm shift.
> i think ddos mitigation is really a place where free software can cause a massively needed paradigm shift.
I'd argue resilient, reliable network gear in general is such a place. Reassuring to hear that Google is adopting Openflow and rolling their own ... maybe that'll trickle down. Cisco gear has led the way of being overpriced for fucking years.
If the packets are random UDP sources, then there's not really much you can do on the receiving end except strategies that involve the destination address. There's really only one effective one: nulling it.
I, personally, have absolutely no need to have incoming UDP packets of any kind at all entering my network, and can not come up with a reason why any web hosting company would: it seems like it should almost be a question on your bandwidth contract "will you need UDP (recommendation: no)". (Given the simplicity of the UDP problem, I personally assumed that the complexity would come from state-ful TCP filtering issues.)
It would be tempting to blackhole UDP, but it's just as easy to flood a pipe with TCP. You don't need an established connection to get packets in the pipe, and there are many mismanaged networks on planet Earth (as I alluded to) that let just about anything exit. A small botnet of five or ten machines with gigabit uplinks on poorly-managed networks is enough to be a force to reckon with. When I was an administrator on IRC, Cisco routers themselves were common targets to exploit and use for this purpose; often, they're directly plugged into gigabit or even ten gigabit connectivity, and there are really easy ways in IOS to perform a DoS attack.
I can't speak to the decision Pastie's host made in this case, but it sounds like since Rails Machine was donating resources (and Pastie wasn't paying them), Rails Machine must act as a smart company -- certainly, any entrepreneur who enjoys Hacker News would sympathize -- and protect its income flow. Which means, you mess with my customers, I fire you.
It seems as though you're skewing the issue here. And I think the real issue has nothing to do with whether pastie.org was a paying customer. I'd be interested to know if RM would do the same thing if there was no sponsorship arrangement and it was paying regular bills. My hypothesis is they'd throw them under that same bus -- and that's really what this comes down to. It's hard to be sympathetic with a company that gives up on its customers (paying or not) after "9 hours". Given that they had been hosted for 3 years prior, a night of DDoSing seems like a really isolated incident, and no reason to drop them permanently. Of course, we don't know if there were other DDoSes, but given that wrecked was so eager to share the piracy concerns and didn't mention any other DDoSes, I don't think there are any.
Nine hours pass. You get frustrated. You take to Twitter. Anybody else on Amazon down? you ask. You get several people to confirm that they are. You tweet that it's an Amazon issue from your company Twitter. You start Googling alternatives. You write a blog post, months later, about how incompetent Amazon must be and you're so glad that you moved your million-dollar app to Rackspace Cloud. You make the front page of Hacker News. Hundreds follow you. Amazon gains a reputation for unreliability among those that read HN. Sales start decreasing.
Or, they null the customer and none of this happens.
Welcome to hosting.
Wouldn't one have to be dropping the packets at the upstream level in order to keep the pipe clear? At larger scales, is it common for the datacenter to own/control a router upstream, or would that be something that they would have to get in touch with their upstream provider to make happen?
It is important to understand that I put our existing customers that pay us to manage and scale their high growth revenue-generating web applications before all else. This is the core of our business and what they trust us to do. As we are seeing now, this means that I will protect them at the expense of making some non-customers and "risky" customers upset. Let me explain further...
Rails Machine at this time is 6 people. Through a lot of tools like Moonshine, experience, and process we manage 100+ web applications. Please note that I did not say "host". Hosting is only part of the package. We commit to do whatever it takes to keep our customers' applications available and growing their business.
Everyone in the organization is a developer on a varying scale of dev to ops including myself who started with Rails in 2005 and have been a professional dev for 20 years.
Although not as quiet as we would like, in general the workload of responding to outages, bugs, scaling problems, and traffic bursts are managed by the team. We've been doing this for 6 years focused specifically on Rails and have seen most problems with Rails applications in production. This makes us fairly efficient in identifying and resolving issues.
We've been hosting Pastie pretty much since the beginning and free of charge for several years. In the past two years, Pastie began to attract a lot of users intending to use it to do illegal things. This includes sharing stolen credit cards, stolen passwords, phishing schemes, copyrighted content, virus/trojan horses, hacker scripts, confidential corporate info, etc, etc.
Please know that the overwhelming majority of Pastie's user base are well meaning folks who kindly follow Josh's basic rule of "using Pastie for good". A tiny minority however attract a lot of attention through their public pastes that ruin the experience for everyone else.
Aside from the obvious problems, the public existence of this stuff makes a lot of people upset who in turn threaten us. This includes but not limited to criminals, giant corporations, angry individuals, trolls, and more importantly our data center/upstream provider. These upset people then send us nastygrams requiring us to take action or else. "Or else" includes suing us, arresting us, DDoSing us, and more importantly terminating our service.
To avoid bad things happening to us and all of our other customers, we have to take action immediately. Every now then other customers get a spam notice or a DMCA notice but in general it happens once and not a huge deal. Pastie on the other hand generates 100s of abuse complaints. Abuse complaints that we can not ignore and require us to investigate and follow up on. While we are doing this, we are not helping the 100+ other customers that generate zero abuse complaints and most likely never will.
Enter the DDos. Over the past 6 years, we've handled multiple DDoS attacks on different applications. Given that 95% of our customers are running revenue-generating business applications, we deal with a DDoS about once or maybe twice per year. They are really annoying and consume a lot of time and a lot of concurrent team members. Although we wish they never happened, as some have pointed out DDoS attacks are a fact of life. Many high profile sites with much larger teams and budgets have struggled for multiple days fighting waves of attacks. We accept this as part of our job.
The problem in this particular instance is that these DDoS attacks on Pastie were a continuation of the stream of operational disruptions already being generated by the site. After handling the first attack with four of us covering all of the angles around 10pm and with the help of Internap's network team, we halted the attack.
Within a few hours, it began again in the wee hours of the morning. At the same time, alerts for another customer who had entrusted us with their business came in. So a decision was made to halt the second attack as quickly as possible and focus on doing our job as we promised to the rest of our customers. We could have chosen to ride out multiple other attacks and engage in a lot of time consuming and expensive behavior to preserve a site that was already a source of ops disruption. Making that choice would have been inconsistent with our values and commitments.
This decision was purely ops motivated to protect our team members ability to serve our core customers.
Some of you are upset about this decision and I am sorry for that. I know 100+ customers that would approve. I put my customers and team first.
Please feel free to reach out to me directly (email or twitter) if you would like to discuss this further.
@bradleyktaylor, Founder, Rails Machine
IMO, your response does more damage to you than trying to explain it away. Doesnt matter if the guy is not paying you or not. In fact, your privacy policy says you wont do this and you just did.
I am not sure whats worse when you get hosted with you:
Is it when they disconnect you for having a incoming attack, or the public post afterwards where they air your dirty laundry?
Wow, its even worse on twitter:
https://twitter.com/#!/bradleyktaylor/status/194937146153508...
If a YouTube employee said that some naughty YouTube users post copyrighted videos, it wouldn't be private information or libellous as such videos are publicly viewable.
I'll edit to clarify that the overwhelming number of Pastie users are "using it for good" as Josh politely requests on the site.
Keep on keeping on. You made the right choice and in the long run dealing with the take down notices and legal wrangling would have been a full time job.
What most people probably wont understand is that no sane business is going to go to bat for a non-customer who is costing them signifigant time and money as well as putting their whole business at risk.
The operational awareness in this comment makes me wish everybody would print it out as an example of a tough call to make in defense of your brand.
Only if the brand is "we're el cheapo hosting cowboys" it's the right decision.
At this point they had to make a choice and they chose for their paying customers and drop pastie.org
- Not being able to handle 100s abusive requests is an indication that you can't handle rapid growth. You're going to (want to) host other websites that can grow big and will catch a lot of wind. Although there is no direct revenue for you Pastie.Org was good practice for this it seems also this wasn't a sudden problem (you yourself indicate here that the problem was slowly growing) so this should be no factor in the decision to terminate the hosting so suddenly. Of course if it's more hassle than it's worth you and Pastie can come to an agreement (like stopping next month) but it now sounds like that decision should've been made months earlier or not yet at all.
- Although it's extremely hard to protect yourself from DDOS attacks you've now openly indicated that you're vulnerable and that you will drop things you are committed to do if there is some pressure in order to save the rest of your customers. This will make other customers nervous since they can now be easily threatened and might even go down for the fun of it (I personally don't think it's fun but some people apparently do)
- Even though pastie.org was not generating any revenue and was not a 'customer' you still had a commitment to them and tbh it looks pretty bad that you've one-sidedly decided in an instant to break that commitment, this is really costing you cred.
Of course what's done is done now, but I just wanted to voice my opinion in this debate. Say I'm a devil's advocate here because by the positive comments here mostly you guys seem to be doing mostly good!
They can play it up, but if you throw enough bandwidth at /any/ host, they'll null route you. Other hosts (at least, the not-stupid ones who have been in this position before) know this and (hopefully) wouldn't sling mud at them over this.
Me too, just out of curiosity, but if you have an attack that is large enough to be affecting other clients, you can't usually wait around too long for the targeted client to respond (and even if they did, there isn't typically much you can do for a large attack.) How long? I'd say about 5 minutes. The typical attack we see rises up over the course of a few minutes and lasts 6-48 hours.
Genuinely curious and haven't really looked around the site much, and you guys hit my radar when you took on Lulzsec. Can't imagine the plethora of nasties directed at their gear.
I tend to use Gist now because its improved and I have a client for it, but this is a shame to see.
``` x = [] ```