An update from Linode about the recent DDoS attacks | Hacker News Reader

An update from Linode about the recent DDoS attacks

status.linode.com

143 pointsalexforster100 comments

Glad to hear something official on this...5 or 6 days is way too long to go without something more than "We're working on it" and some light details. I understand that it's likely an all-hands-on-deck hair-on-fire situation over there, but those of us who rely on Linode for our own businesses have been largely left in the dark.

When our customers are emailing and tweeting us and they just want to know when we are going to be up, and all we can say is "We have no idea, we don't know why this is happening or what's really going on", that's pretty much the definition of a worst case scenario from a customer service standpoint.

As someone whose business relies on Linode currently to function, I am sympathetic to Linode's plight...this is the equivalent of someone coming and setting off a bomb in your factory; not exactly something that you can always plan for even if you have prevention measures in place. But they would have kept a lot more of my sympathy long-term if they would have communicated better with their customers in the first place...

EDIT: And it looks like the attackers decided to start things back up again, as Linode.com is unavailable...

"It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business."

The timing of the DDoS was pretty interesting too, happening when not everyone is available.

This was my first thought when I woke up to a bunch of my server monitor alarms at about 4am on Christmas day....

We know that we've dropped the ball here. To be frank, it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent than "they're attacking our webservers", "they're attacking our core routers", etc.

> And it looks like the attackers decided to start things back up again, as Linode.com is unavailable...

They're watching our status page for updates and starting new attacks when we resolve previous ones. There's been an almost 1:1 correlation lately.

Indeed, well, keep fighting the good fight. We are cheering for you.

Thank you for the update. We really appreciate the information and wish you the best of luck in dealing with this.

I just registered to thank you for the update and to let you know I won't be moving away from Linode as this just gives the attackers what they want.

The correlation could also be the fact that you've blocked the attack, they can see that their previous attack is no longer working by testing it themselves - so they'll switch anyway, regardless of a status page update.

> it's just been extremely difficult to take our people off of mitigation long enough to write something more coherent

This always rings hollow to me, and yet I hear it over and over.

A company like Linode surely has at least a dozen people who can be on call in a situation like this, probably much more. All it takes is for one engineer or even a product person... Heck a technically-minded support person could listen in on the war room meetings and get enough information to post something better than "we're fixing it".

5 minutes of blogging every six hours would be plenty.

And yet, people always claim it's impossible and there's no time. Frankly I find it frightening... If you are coding that fast that no one on your team has five minutes to step aside, take a breather after six hours of coding and summarize what the team just spent the last six hours doing, I shudder to think what kind of panicked alarmist interventions you are making.

There's no excuse for silence. There just isn't. It's a gross failure on the part of management to prioritize the responsibilities they have to your customers customers. Full stop. Sure, the engineers can't be expected to remember to tap out to blog. But if that's the extent of the accountability structures you can assemble during a crisis, that is a serious organizational failing, particularly for an organization the size of Linode.

I would say that 5 minutes of blogging every 6 hours is just as likely to cause more harm than good. Wording something badly could lead to far more fear and confusion than the present situation if taken out of context.

> As someone whose business relies on Linode currently to function

Perhaps its time to consider some failover at another host. Same goes for anyone solely dependent on anyone.

Certainly what we're in the process of doing.

I'm surprised you haven't completely jumped ship already, considering that there are two other providers in particular that offer very similar specs and prices to Linode. Where I work, we moved our primary infrastructure off of Linode the night after the attacks started.

I've tried a lot over the years, no one can compare to the VPS quality Linode provides is what I found.

All (major) providers suffer from DDoS attacks.

When was the last successful sustained (1 week plus) CloudFlare, GCE, or AWS DDoS attack?

I have no idea.

CloudFlare doesn't offer VPS instances. Google and Amazon are in a league of their own and, last I checked, don't offer the similar services at the same prices with the same level of support and same controls. Amazon might send you a nice bill at the end of the attack depending on what type it was.

DigitalOcean, as an example, routinely receives DDoS attacks and will just nullroute your VPS automatically until the attack ends.

There are VPS providers that offer DDoS protection in the 10-20Gbps range, but that is a piddle in the ocean these days.

Which ones and protection against what attacks?

Ramnode, Vultr, others.

Its a concern of mine with AWS, no downtime but a $30k for the day of fun bill. Got to tie up this loose end before I wake up broke one day.

What's the current status for Database failover? MySQL in my case - to mirror that is what's been holding me back.

What happens with 1/10 second's database records not replicated? That could be a successful transaction on one, and no record of it on the other?

> What happens with 1/10 second's database records not replicated?

You can use a multi-master setup and strict consistency checks, though.

Multi-master MySQL is generally regarded to be a very, very bad idea. Single master with multiple GTID replicas and promoting a replica to master upon failure is BCOP.

How do you guys re-route traffic for your load balancer to another in another data center? If you change your DNS record it would take x time before it has propagated.

Say I have one server setup on Linode London and one duplicated on DO Amsterdam. How do I quickly re-route the traffic from my two main load balancers London to Amsterdam without much propagation time?

You cab use round-robin DNS ... Keep two DB in sync between different hosts/providers and effectively switch the DNS if host1 or host2 goes down. However this setup is easy for a kind of applications but may not be suited for other kind of apps.

Keep TTL's low (3-5 minutes). Things don't have to happen instantly. Clients understand. You had a bad day on the Internet. Guess what, stuff happens. Remember when "XYZ company down for 2 hours" made headlines? Yeah, that was a long time ago. Downtime is more or less accepted these days. Sure, some hothead customers will flip out and leave. You were probably not going to meet their expectations sometime and lose them eventually. Let them become someone else's nightmare.*

* This is for an answer for the average "Hey I run a website or two" question. If you're asking and you are dealing with financial transactions or ad networks, well, you're asking in the wrong place and you should probably hire someone that knows how BGP anycasting works.

>* This is for an answer for the average "Hey I run a website or two" question. If you're asking and you are dealing with financial transactions or ad networks, well, you're asking in the wrong place and you should probably hire someone that knows how BGP anycasting works.

Eh, usually you just bgp anycast the DNS server. If you anycast just your DNS server, you are pretty much in the same pickle as if you host your dns with someone else who runs a reliable dns server (and have them do the anycast) - the pickle is that there's a minimum ttl.

Now you can set up a redirector server and anycast that, and like 3xx redirect all your requests (or more often just the first request) and that's better, really, 'cause that redirect can be changed quickly and on a per-request (or more often per-session) basis... but while I know a thousand companies that will sell you anycast DNS service, I don't know anyone who will sell someone with a vps customer budget an anycast'd redirector service.

We use a CDN. They provide caching and act as another layer of defense beyond the origin datacenter/provider.

Because they physically route the traffic, you can change backends pretty quickly and have traffic going to the new destination within seconds or minutes. Using DNS for end-users just has too much unpredictable latency, but it works well for routing the backend.

[Disclaimer: Linode employee]

Around the holidays, network engineers are the only ones who don't really take time off. The sorts of people who might say "hey, we need to give some clarity to customers" are less available than the people whose time is spent firefighting.

Can you share with us the mitigation techniques you tried so far and what worked better, what did not work at all, etc?

Stopping them from directly hitting our routers has been the most difficult, because our routers have IPs on 1000+ /24's, which means RTBH can't work because no tier 1 will accept 1000+ null routes. The same is true when they attack our upstream provider's networks.

We've had a few false-starts, but we've finally gotten all of our datacenters dropping all traffic to these critical IPs at the edges. This should stop the most serious outages, so it's a big step.

As a happy linode user, your update is much appreciated.

I do feel a certain amount of loyalty to linode, as they've been an excellent service the past few years, and I see them battling to put out these fires.

I'm hoping as linode grows in size they can put in place more sophisticated measures to guard against this - DDoS is a problem everyone faces. Its tough, but when the dust settles, it could be an opportunity to innovate.

Stopping DDOS isn't so much about innovation as cost: it's metric tons of data thrown at points you control and don't which have to analyze and react to it somehow. That's assuming it doesn't straight-up fill the pipe where ignoring it changes nothing. Here's a case study showing how expensive DDOS mitigation can be for a smaller firm:

https://protonmail.com/blog/ddos-protection-guide/

I think "whole data center is offline" is the sort of thing you call the boss for. Otherwise why is there a boss at all?

I've seen Linode getting a lot of flack for not updating customers on what is happening, but what clarity does this announcement provide that their DDoS status page wasn't providing? They were updating the status page regularly.

Push, not pull. Send an email. Don't expect people to go to a status page on their own.

Which they could easily do by enabling SMS/email/webhook notifications in their statuspage.io settings. As it is, you can only subscribe to an RSS feed it seems.

It gives those of us hosted on Linode a clear place to point our customers.

I can't believe people are criticising Linode:

1 - Attack mitigation was mostly successful. As I thought and they have confirmed, the attack vectors evolved continuously.

2 - They had to deal with this over Xmas. Anyone familiar with such a job knows what this means in terms of human resources, knowledge distribution, organization of technical response and communication with 3rd parties.

3 - Linode is not Nagios. If you don't monitor your own infrastructure don't expect Linode to SMS you because your site might be down. Linode resources were focused on fighting the DDoS, as they should, and provided regular updates through their status site, as is expected. Everything else is nice-to-have, but no a must-have.

4 - In line with what others said, I had 7 hours downtime in my London VPS. That is an uptime of 96% in the last 7 days. Considering restless DDoS ongoing over holidays, I'd say that is pretty good.

I'm sorry, but what happens to Linode sucks, but it is an eventuality anyone with assets depending on this service should have counted with, because it can happen everywhere. Cannot blame Linode if your HA strategy does not exist, or you never thought of a way to gracefully fail over to a second provider if your business depends on >96% availability.

1) They were caught with their pants down. Their DDoS playbook was probably either outdated or not full fleshed out. This is somewhat excusable if you're a content provider. It's not excusable if you're the cloud/colo/datacenter provider. This is literally your raison d'etre. Cue the "you had one job" memes.

2) Netops people don't get Christmas off (1). Our teams are working all year, all day, every day. Netops isn't HR or marketing. It's not super difficult to arrange a conference call with your transit providers, DDoS mitigators, etc. This is old hat to them.

3) Completely agree.

4) The downtime was a lot worse if you had multiple instances in more datacenters. Or put another way, the bigger the Linode customer you were, the worse it was.

(1) I've had a router die every Christmas or New Years for the past 6 years. Never had one die outside those windows. They must angry at me for what I've put them through.

My message to the attackers (in case they happen to read HN):

Fuck you. I will continue to be a Linode customer. Not sure what your goals might be but you will not succeed.

Frankly, and I am going to be politically incorrect here, these are the kinds of cases where I wish there was a "special forces" kind of task force to hunt down these pieces of shit and put them out of their misery.

This amounts to financial terrorism of the worst kind. It affects small and large companies and creates untold losses across the board. It is entirely unproductive. The world would be a better place if the pieces of shit who engage in this sort of financial terrorism simply didn't exist.

Happy New Year.

Linode folks: I'm renting another server next week. Don't need it. Just want to support your effort and, in a tiny way, help mitigate losses. I might just give it to the kids in the robotic team I mentor so they can play around in a real server environment.

> Frankly, and I am going to be politically incorrect here, these are the kinds of cases where I wish there was a "special forces" kind of task force to hunt down these pieces of shit and put them out of their misery.

Oh, it's certainly in the capability of the FBI and the NSA to hunt them down. Even child fuckers inside the Darkweb got busted.

The problem is priority: unless either child porn or a huge US company is involved, the three-letter-agencies don't give a shit about this kind of crime.

Which is a shame, because some action would probably dissuade future attempts.

This is NOT the fault of the attackers. The continued efficacy of DDoS attacks is squarely the fault of ISPs who absolutely refuse to police compromised customers or filter egress.

This is a problem that has a solution, but the people in the best position to handle it have shown no interest in doing so for a decade.

Um, yes it is the fault of the attackers. Attackers aren't animals or clouds. They are agents that have free will and are choosing to try and destroy Linode, probably as part of some sort of ransom attempt.

ISPs can't be expected to solve this problem: the ones that have tried discover that it takes huge efforts to get a hacked customer cleaned up, and then they frequently just get re-infected. ISPs exist in a highly price-sensitive and competitive environment: providing dedicated, personalised security services to customers who don't even care or can't be blamed (e.g. hacked router) is incompatible with charging a competitive price.

I don't know what the solution is. Just saying "no" next time someone wants to use crap programming languages and frameworks that systematically lead to exploits would be a good start. No more using C in embedded devices that can't be remotely upgraded.

If you trust people to do the right thing, you're going to get burnt.

We need to evolve the internet past these types of attacks and build systems to protect everyone from this kind of terrorism, rather than just get angry at people who abuse well known about loopholes.

Your anger, while justified, is not helpful.

You are absolutely correct on all points. Yet people need to get angry in order to be motivated enough to cause change. Happy people don't push for change, not on these kinds of issues. There has to be a lot of pain felt and verbalized for anyone to take it seriously.

Example: US government and all necessarily parties start to shut down --as in kill off-- all packets coming in from China every time there's a large DDoS attack. If it stops, we know the source. And we treat them to a full day of packet blocking to see if the Chinese government might become inspired enough to do something about it. Next time, it's two days. Or maybe a week.

This is the kind of thing where pain has to be felt at a high enough level for those who can do something about it to intercede and put an end to it.

And, yes, in the end, it's a technological solution. I won't claim to be informed enough to fully understand what has to be changed in terms of protocols, hardware or topology in order to make DDoS a thing of the past. I simply haven't spent any time studying the problem. I'm sure there are lots of people very well versed in the subject who can pitch in.

At some level though, I think this has to be criminalized and prosecuted with great consequences for the perpetrators in order to put an end to it. I say this because it is likely that whatever new technology is instituted there will be holes bad actors can exploit.

Thanks for the update and the hard work. You all work a job that requires a lot of sweat and tears that goes unappreciated from many levels of our society. Making the internet work is hard work.

Know that you're in good company here and that we're rooting for you.

We appreciate this more than you know. Many of us have had the holidays ruined by these utterly relentless attacks, and it's a difficult thing to try and explain to our loved ones. Support from the community really helps.

When I noticed this on Christmas Day (and saw my suspicions confirmed on the status page), my first thoughts were along the lines of "Wow, that's going to ruin Christmas for a lot of Linode families this year." My heart sunk for you guys. I can't even begin to imagine the stress it's placing on friends, relatives, and immediate family members. (The inevitable "But it's Christmas..." comes to mind.)

I just want to say that I switched this summer to Linode from DO for my personal sites, and I've been very happy with your services. I plan on launching a few more things in the coming year, and these attacks have galvanized my resolve to continue supporting you through all of this. Thanks for everything you guys are doing, and keep up the good work!

I know it's unlikely to ever happen given the nature of these attacks, but I really do hope the perpetrators are found, brought to justice, and locked away for a long, long, long time.

Tell them to imagine a busy, but orderly, grocery store where people are getting what they need for New Year's meals. It's one of few places community depends on to get what they need in life. Then, mask-wearing mobs of 500+ people rush in using their bodies to block every line, aisle entry/exit, register, and store exit. They physically stop all traffic from moving despite screams of protest and nobody can even call the cops because they jammed that, too. People squeeze by a little here & there but all action is frozen. And they pull this stunt on Christmas and New Years for a week straight.

Tell them that. Then say what happened to your business and customers was the digital version of it. They should get the idea.

+1 to this. I've been a happy Linode customer for 4-5 years now. The few times I've had to use support they've been great. I also appreciate the regular upgrades to their offerings meaning there's little need to shop around and hop hosts. The ddos attacks are annoying, but I will remain a loyal, happy customer.

My London Linodes' stats from the past 7 days: 40 outages. 6h25m downtime.

What upsets me the most is that customers haven't heard a single word from Linode. If you weren't watching their status page (and have server monitoring on your servers), you'd be clueless.

The least they could do is to email affected customers about what's happening, and the time frame they need to fix it.

But one week of continuous issues is just not good enough.

That being said, I love Linode, and will not use any other primary provider for my servers. They have been dead stable the past years before this happened, and never had any bottlenecks (e.g. overselling servers like other providers). And I feel sorry for the network engineers trying their best to fix this. The missing information is a Linode Customer Relations issue, not the engineer's.

Happy new year to everyone. And looking forward to a great 2016 at Linode.

1) Why in the world are you exposing your router control planes to the outside world? That should be ACL'd off (in stateless firewall rules and routing engine) to only allow access from a few IP's.

2) Your transit providers should be defending their infrastructure. I've never seen a transit provider allow an attacker to take out their /30 serials or IX addresses. This is their network after all. If attackers try to hit the serial between customer and provider, you just readdress the serial to RFC1918 space. You don't really need a routable address there other than to make traceroutes easy to read. If they attack farther upstream in the provider's network, you just add ACL's at the provider edge. Nothing external will ever need to reach a provider's core. This is basic, basic stuff.

Next time, don't only run your network on house bandwidth (HE, TelX, etc). Or in other words, caveat emptor.

I want to cut Linode some slack they have been great but those were my thoughts when the employee above divulged that info. It all seems a little like using cold fusion to run your whole shop..

People dont need to jump ship they need plans in place to deal with problems like this, even just a rsync.net account.

Judging by the downvotes I got, there are people on HN that must think you should expose your router's control plane to the Internet. That or they expect transit providers to not protect their network.

There's no intelligent discussion left here. It's devolved into a sympathy vote. "You said something mean about Linode. I love Linode. Downvotes for you."

If Linode had posted the update to NANOG, it would have been more productive. I don't often say that.

You may want to go pinging around some of your own tier 1 crossconnects. I bet you'll be surprised.

They all respond because I asked them for it. About half didn't upon turn up. It's super trivial for them to null route them or readdress in rfc 1918/4193 space.

Or are you referring to xconnects inside their network? That's up to them to work out and I've never seen a provider just abandon their network while under attack.

You may want to reconsider. CoPP only goes so far, as we've learned the hard way.

I don't run Cisco, but thanks for worrying about that. Also, stateless ACL's should protect against overrunning the control plane.

Link to original thread, which has been getting comments up until today: https://news.ycombinator.com/item?id=10806686

> a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business.

I wonder what size investment this is taking, and what the end-game is for the bad actor. Unless Linode's mitigation tactics are increasing the bad actor's costs, what's to stop the bad actor from continuing the attacks until Linode goes out of business?

.. it may play out that the best defence is a good offense - ie. to hunt down and attack the attacker.

I imagine there are a few thousand linode fans who'd be happy to help fight back.

Stolen credit cards go a long, long way in AWS and GCE capacity.

Exactly. Plus, botnet capacity was cheaper than AWS last I checked where it was around $1 a machine. So easy to pull this off on the cheap that it could be anyone for any reason.

Greetings.

I know I migth come late, but being one of your very satisfied Customers, and having experienced this type of issue multiple times with other providers that didn't even bothered to even ackowledge that there was an issue, I can say that I wil remain with you regardless.

Also to those saying "I have all my business running at Linode so this is unacceptable", I only says this: You get what you pay for, and for a VPS Service you won't find better than Linode, and if you have something critical running for YOUR Clients, than it is YOUR responsability to ensure resiliency against this type of situation. Linode is a VPS provider after all, and the reason why you are making money out of someone who doesn't know enougth to go to the VPS hoster themselfes.

Good luck making a profitable business and milking your Customers running on AWS or AZURE. You'd be broke and in debth at the first DDOS and Over-Bandwith charge from any of them.

I work at a service provider myself, and I understand what you guys had to deal with the last 10 days, and you have my full support.

I'm a bit surprised to see the update from someone who is not senior management/C-level (as far as I can tell). Where is the communication from the CEO/CTO?

It seems a bit unfair to have this fall on Alex's shoulders. I could be way off base, happy to be put right. I'm sorry to hear about your ruined holidays. Hopefully you'll get some time off soon :)

That's crazy... I have a linode box with several low-traffic websites on it, old projects I've wanted to keep around for archival purposes. I picked linode because I wanted root access and they were cheap but really just because I wasn't sure of better options. I suppose there is always t2.nano.

Don't forget that SSD and Bandwith is at additional charge at AWS, so it will be a lot more expensive than DO, Linode or Vultr. (Also doesn't include Support costs, if you need it)

You get what you pay for. t2.nano isn't as fun as it sounds. Just wait until someone soaks up all the available instances surrounding yours and starts generating CPU steal on your box, thus making everything in your OS take 50-90% more time to complete. I've seen this happen on t* and m* instance types and it's especially the case when they're the smaller sizes in that instance class.

Linode has always been a great host. Sure they've had their growing pains but I've never been more happy with a virtual hosting provider, even their support. But yes, days without communication is not a good thing. Let's hope they learn from this.

ISPs already spend plenty of money on DPI and HTTP injection gear. It would cost next to nothing to do basic egress filtering and detecting+throttling known compromised customers.

And yet, we still get DDoS attacks. Why?

Because these days "botnet" can easily mean "botnet of compromised Linux servers" or "botnet of WiFi routers". The biggest DoS attacks are often based on exploiting UDP based protocols and so you end up being attacked by ISP/university-sized DNS or NTP servers.

I think the lesson here is don't rely on one supplier. I have my tiny infrastructure spread over three different suppliers in different geographical locations. Plan for the worst and hope for the best.

Edit. This is in no way a criticism of linode. The worst outcome is if we all end up with one monopoly supplier. I have deliberately avoided using the big player in this space as I want support diversity. This makes my job harder, but it is better for us all if we don't put all our eggs in the one basket.

Have you done a write-up of how you've managed to keep a real-time sync of assets/database? Or are you hosting static websites?

This is what I'd like to know too. It's easy to suggest mirroring your infrastructure on multiple providers but in practice it's very difficult. Multi-master SQL in particular is non-trivial and that's just one piece of the puzzle. We have MongoDBs, CouchDBs, Redis, and Elasticsearch -- each with varying difficulty and costs to cluster.

You're definitely not mirroring your infrastructure any time soon :). Not to say that it's impossible because I don't believe in impossible, but that's quite a stack to replicate.

We have much the same stack at Reverb.com and I focus more on creating a disaster recovery plan than tackling a mirror of our infra. I'm focusing on bringing up our stack in another region of AWS if we would ever need to rather than mirroring everything.

Sometimes a mitigation plan is far more useful than building in excessively complicated redundancy.

That's interesting and probably more in line with what I need to do. It wouldn't be difficult for me to bring the entire stack online in another datacenter, but if all of your customer data resides on (for simplicity) a master SQL node in Linode-Dallas and that datacenter goes offline, how can you bring that backup node online when you can no longer access the Dallas machines?

Are you suggesting to have a slave ready to become master in the 2nd datacenter? If that's the case, my main area of uncertainty is: if you promote a slave to master in a 2nd datacenter and that new master accepts writes, you then have a brutal problem of propagating writes to the old master when he comes back online.

The whole thing may exceed my capability right now but I think having a disaster recovery is far better than a mirror now that I think through it.

I'm curious about the stuff that's hard to replicate. Don't most of them store their stuff in files? Could those users perhaps use a distributed filesystem for replication? Clustered filesystems were used in supercomputers, are used by Google, and similar methods go back to VMS systems on VAXen.

I'm not saying something off-the-shelf will work or out of the box. Just seems like there would be a clear path for admins or developers of such software to integrate it with... something [1]... that did that.

[1] Example: http://sector.sourceforge.net/

Nobody said anything about multimaster. You keep a hot standby in a different availability zone and stream your transactions to it. Failover when needed.

Unless you're a bank, you run asynchronous with watchdog scripts and accept the very very small risk that you lose a few transactions within the ~100ms window of async latency.

It's all risk analysis. For most of us, hours of downtime is riskier than losing a tiny sliver of data for a small number of customers.

That's a proven approach going back decades. A number of banks do it, too, because it often works well enough in practice where data isn't lost. A trick on that is having something on each end that sort of checksums the data(base) for certain time periods to see if anything was lost. Then, resends that stuff. My use of that was inspired by the transport layer of the Tsunami file-transfer protocol with the implementation very similar, too. :)

I haven’t done a write up, but basically I just use unison [1] with a hub and spoke model. It is pretty simple, but it works well for my use case which is all file based.

My system is dynamic, but once a user is allocated to a node it is easy to replicate their data over the network. The key was I designed the system with easy replication in mind - flat files rather than databases, etc.

1. https://www.cis.upenn.edu/~bcpierce/unison/

Then what's your take on my suggestion?

https://news.ycombinator.com/item?id=10822904

EDIT to add: Good work on Unison btw.

I think it is a good idea. It is a bit more complex than using Unison, but it should be more generalisable.

I think the real key to creating a simple and low cost reliable system is to have this in mind when you design your application. I could have used a database just as easily as a flat file approach, but this would have made things much more complicated to replicate between the nodes.

I appreciate the feedback. This...

"I think the real key to creating a simple and low cost reliable system is to have this in mind when you design your application. I could have used a database just as easily as a flat file approach, but this would have made things much more complicated to replicate between the nodes."

...is a great point. It echos the comments and thinking of Bernstein in his paper on lessons learned from Qmail.

http://cr.yp.to/qmail/qmailsec-20071101.pdf

Particularly, section 4.5 "Reusing the filesystem."

This isn't necessarily directed at you, but I frequently see products which have no disaster recovery because they can't get something that seems good enough, contrary to any economical reasoning about what "good enough" is.

It doesn't always have to be complicated. Chances are It'll probably feel more like grunt work than rocket science.

Look up Disaster Recovery Metrics. You set an RTO (recovery time objective -- how long it takes to recover) and RPO (how much history is lost when recovering) based on what's needed for your project and feasible to implement.

Feasibility of disaster recovery strategy = Cost to implement strategy + Cost of losing data back to the strategy RPO + Cost of losing business during strategy RTO < Cost to business of riding out probable disaster

Pick the fastest/cheapest feasible strategy, implement it. Then once your ass is covered, pick the one with the greatest expected value and implement that. If you wait until you can design and are given resources to implement a 5-nines available strategy with sub-milisecond RTO and RPO (which seems to be a popular tendency of developers), disaster will strike before it happens.

Great advice, thanks. I am one of those developers who chases after "a 5-nines available strategy with sub-milisecond RTO and RPO" - because that's what the suits want, and I get treated like an idiot in meetings where I say it's not possible/not cost-effective.

And, y'know, because it would be really cool if I managed to ;-)

Try this. Tell them there are three models that regularly achieved five 9's with low latency for over a decade each. Each with clustered or HA configuration over dedicated lines. List them as such:

1. IBM mainframes.

2. HP NonStop systems.

3. OpenVMS clusters on SMP machines.

Give them the prices of systems and rare labor for each. Then, show them the price of a three to four 9's solution with Linux that has three to four less 0's. Mention how many extra bonuses, err business investments, they could make with such savings. At this point, they might treat you like an idiot for not buying the cheaper route. Maybe even enthusiastic about your business savvy approach. ;)

Out of curiosity: why is it so hard to track the real origin of DDoS attack, who's behind them and what they are after?

I'm willing to bet it's someone with money who has an interest in making Linode look bad or unreliable. I can't imagine someone would sustain an attack like this for shits and giggles.

Imagine the scenario: person A sends a malformed DNS request to a bunch of DNS resolvers, asking them to send the response to person B. Now imagine that person A is actually part of a large botnet, being controlled by person C, via some smoke-and-mirrors.

If you're person B (under attack) it's pretty difficult to track through all of that to person C. You'd need a lot of cooperation from people (likely in many different countries) who really just want to go back to their normal business. They're likely also charging for the traffic, so they're not really that bothered, and they're each only seeing a small proportion of what person B is seeing so they don't see it as much of a problem (so aren't likely to be inclined to get involved).

How do botnets still exist? Wasn't that a problem that should have gone away with Windows XP?

Windows XP hasn't gone away.

So are the botnets that still exist mainly running on the legacy XP boxes?

It says here that it's roughly 10% of machines. It's safe to assume at least half are infected. But aren't they mostly third world?

https://www.netmarketshare.com/operating-system-market-share...

Android too.

https://encrypted.google.com/search?hl=en&q=android%20botnet...

Devices running Windows XP aren't the only ones that can be compromised. Other possibilities are unpatched Linux servers, servers with easily guessed root passwords, consumer routers, etc.

jsonip.com is hosted on Linode. It's been averaging roughly 6mb/s inbound for months, but in the last week it's been about 8.5. I'm not sure if the uptick has anything to do with the DDOS attacks or not.

Did you also run jsonip.org? I used that occasionally (not programmatically, just when I wanted to see my IP) and it started returning a 502 a few months ago.

Reply on news.ycombinator.com