Also, why did Rails Machine throw out the site so quickly? If I choose to sponsor someone out of my free will, I'd do so without distinction from paying customers.
>"why did Rails Machine throw out the site so quickly?" If you run a datacenter, you pay for an uplink. That uplink has limited capacity. 4gbit, 10gbit...whatever. A big attack can saturate that link completely, so even with the biggest most expensive "mitigation device" on the market (some of this gear can get into the hundreds-of-thousands-of-dollars for /one/ device, mind you), if a DDoS is overloading your upstream bandwidth providers, you can either have your entire DC brought to a crawl, or null route the site.
With that said, how did CloudFlare keep lulzsec up? Anycast, lots of iron, lots of smart technicians, and probably tens to hundreds of thousands of dollars in bandwidth fees. TL;DR it was a publicity stunt that they very smartly played up.
DDoS is pretty misunderstood, and lots of clients think that there is some magical box that can take all the traffic. Again, if your link is saturated, a "mitigation" device can only filter the traffic; your upstream providers can and will take you offline if you don't fix it. Failing that, you get a massive overage bill and every other client at the facility is crawling. It's not really a good solution (mitigation devices /can/ help with smaller attacks, but for the big stuff, null routing is the best solution unless you have something like CloudFlare -- and even they will pull the plug if the attack gets too heavy, because it's simply not worth the expense to them to keep your site online.)
Edit: Let's say you run Joe's Web Hosting. Joe's has three facilities, and you run redundant ten gigabit uplinks at each. Last I priced a device that could handle ten gigabit at line rate, it was ~$120,000, so figure:
$120,000 x 2 uplinks x 3 facilities = $720,000
(I honestly don't remember if that figure was for the gigabit device or the ten gigabit device. I think the ten.)
network equipment vendors like to have high list prices in order to:
- ensure they get sales inquiries from serious buyers;
- give sales representatives a large amount of leverage when negotiating the actual line-item pricing for each purchase order;
- promote some kind of enhanced "value" to executives ("it costs a lot of money, so it must be good!")
a realistic price for two routers with two 10gig-e linecards is about $15,000-$25,000 per unit direct from brocade. cisco would be around the $35,000-$50,000 per unit ballpark. if you're paying any more than that, you're getting extremely shafted.
further, the circuits for the uplinks aren't going to cost that much themselves, as it's all sold under percentile-billing (not even 95%, usually more like 90% or 80% at over-gig-e levels these days) when we're talking actual transit costs. commits for "Joe's Web Hosting" might only be 1.5gig on each 10gig circuit, so that cuts down pricing a lot right there.
in terms of getting an experienced engineer, most attacks can be detected and mitigated automatically using netflow/sflow/ipfix flow analysis. i wrote software which handles the detection automatically, with a fairly high success rate, and it's free to use on bitbucket. :)
I worked for a serious buyer when I had this casual conversation with the vendor. You do business with my former employer (I've spoken to you before, when you took over cia.vc). I'd consider those facilities "real" datacenters, and they're certainly the same size, if not significantly larger, than your hosting outfit. No need to appeal to authority with me, honestly, I think we're on the same page -- I think your comment misread me as pricing routers instead of mitigation gear.
As for flows, I was interested in a solution that I didn't have to write software to implement. I'd rather have a supported appliance that can handle figuring out DoS attacks itself, rather than me parsing flows and feeding that information back. That way, if the software doesn't work, I can blame somebody else rather than me. My time is precious. Yours sounds less so, and that's your prerogative.
in terms of mitigation hardware, all of those appliances are a ripoff. the solution is to replace them with free software that does automatic analysis on the flows and then sends that data automatically elsewhere, putting them out of business.
i have a system set up that does automatic mitigation using ddosmon by having a "scrubbing center" running on freebsd-based devices. this is accomplished by having a custom 'action' module in ddosmon which does three things:
- calculate the necessary ACLs
- insert them into the appropriate pf tables so the mitigation strategy fits my ruleset
- direct the router to send traffic for the IP being flooded to the scrubbing appliance
this is basically the way that the mitigation appliance vendors tell you to do it if you're handling >10gig floods anyway.
i think ddos mitigation is really a place where free software can cause a massively needed paradigm shift.
> i think ddos mitigation is really a place where free software can cause a massively needed paradigm shift.
I'd argue resilient, reliable network gear in general is such a place. Reassuring to hear that Google is adopting Openflow and rolling their own ... maybe that'll trickle down. Cisco gear has led the way of being overpriced for fucking years.
If the packets are random UDP sources, then there's not really much you can do on the receiving end except strategies that involve the destination address. There's really only one effective one: nulling it.
I, personally, have absolutely no need to have incoming UDP packets of any kind at all entering my network, and can not come up with a reason why any web hosting company would: it seems like it should almost be a question on your bandwidth contract "will you need UDP (recommendation: no)". (Given the simplicity of the UDP problem, I personally assumed that the complexity would come from state-ful TCP filtering issues.)
It would be tempting to blackhole UDP, but it's just as easy to flood a pipe with TCP. You don't need an established connection to get packets in the pipe, and there are many mismanaged networks on planet Earth (as I alluded to) that let just about anything exit. A small botnet of five or ten machines with gigabit uplinks on poorly-managed networks is enough to be a force to reckon with. When I was an administrator on IRC, Cisco routers themselves were common targets to exploit and use for this purpose; often, they're directly plugged into gigabit or even ten gigabit connectivity, and there are really easy ways in IOS to perform a DoS attack.
I can't speak to the decision Pastie's host made in this case, but it sounds like since Rails Machine was donating resources (and Pastie wasn't paying them), Rails Machine must act as a smart company -- certainly, any entrepreneur who enjoys Hacker News would sympathize -- and protect its income flow. Which means, you mess with my customers, I fire you.
It seems as though you're skewing the issue here. And I think the real issue has nothing to do with whether pastie.org was a paying customer. I'd be interested to know if RM would do the same thing if there was no sponsorship arrangement and it was paying regular bills. My hypothesis is they'd throw them under that same bus -- and that's really what this comes down to. It's hard to be sympathetic with a company that gives up on its customers (paying or not) after "9 hours". Given that they had been hosted for 3 years prior, a night of DDoSing seems like a really isolated incident, and no reason to drop them permanently. Of course, we don't know if there were other DDoSes, but given that wrecked was so eager to share the piracy concerns and didn't mention any other DDoSes, I don't think there are any.
Nine hours pass. You get frustrated. You take to Twitter. Anybody else on Amazon down? you ask. You get several people to confirm that they are. You tweet that it's an Amazon issue from your company Twitter. You start Googling alternatives. You write a blog post, months later, about how incompetent Amazon must be and you're so glad that you moved your million-dollar app to Rackspace Cloud. You make the front page of Hacker News. Hundreds follow you. Amazon gains a reputation for unreliability among those that read HN. Sales start decreasing.
Or, they null the customer and none of this happens.
Welcome to hosting.
Wouldn't one have to be dropping the packets at the upstream level in order to keep the pipe clear? At larger scales, is it common for the datacenter to own/control a router upstream, or would that be something that they would have to get in touch with their upstream provider to make happen?
On the other hand, it's also possible that the originators of the DDoS are angry about something the hoster did take down. Perhaps they'd put up material against the rules, Pastie took it down, and in a tantrum after-the-fact, they decided to retaliate with a "well if you won't host our stuff we'll block eveyone". (This is a bit more like the Anonymous DDoS against payment services they didn't like.)
Without a statement from those involved, it can be hard to determine DDoSer motivations, but those are some common patterns.
The attack might have been launched for the fun of it, too. Why not, when you won't be affected personally?