Posterous is being DDoS'd | Hacker News Reader

Posterous is being DDoS'd

twitter.com

84 pointssuperjared45 comments

We're getting about 500k packets per second, 500mbps to 1.5gbps peak, it's a synflood from a botnet. Typically we can IP hop and null-route the old IP's. That usually buys us about a day until the botnet phones home to get the new IP's, at which point we just hop again. Since our DNS TTL is only 5 minutes at most we are down 5 minutes.

TODAY, the attackers hopped IP's to our new IP immediately. So they appear to be learning. But then again, so are we.

Gigenet's anti-DDoS service has helped us a ton here and is now serving as our front door IP to block the synflood. They've been really responsive.

Any suggestions for customers with DNS names pointing to posterous blogs? Are we going to need to follow you on each hop?

Custom domains are unaffected, no action needed.

I wouldn't think so...

Do all the caching name servers respect the TTL though?

They better. So far so good.

I really wish setting DNS TTL's to 5 minutes meant that browsers and everything else (OS caching) would respect that but sadly in my experience it doesn't.

Surprising that it works so well for you.

Gigenet's service looks interesting. They don't provide pricing though. I'm guessing it's not cheap? Is it based on the bandwidth they'd have to use?

Relying on a DNS TTL of 5 minutes doesn't work everywhere. Some ISPs will forcefully reset any TTL below 24 hours to 24h.

AT&T/sbcglobal being one.

Absolutely true, and completely ridiculous. What, DNS bandwidth is too high? Please.

Well, not following the RFCs is bad, but I can see why they do it.

Most people using low TTLs probably don't know what they're doing, and if you're a big ISP having to constantly make recursive queries hurts page load times for your customers, who'll blame you.

This is hot off the heels of an email sent out last night:

As you’re no doubt aware, Posterous has had a rocky six days.

On Wednesday and Friday, our servers were hit by massive Denial of Service (DoS) attacks. We responded quickly and got back online within an hour, but it didn’t matter; the site went down and our users couldn’t post.

On Friday night, our team worked around the clock to move to new data centers, better capable of handling the onslaught. It wasn’t easy. Throughout the weekend we were fixing issues, optimizing the site, some things going smoothly, others less so.

Just at the moments we thought the worst was behind us, we’d run up against another challenge. It tested not only our technical abilities, but our stamina, patience, and we lost more than a few hairs in the process.

I’m happy to report Posterous is at 100% and better than ever. Switching to a new data center will help us avoid the type of attacks we saw last week, and the new, bigger, beefier servers will speed up the site and increase capacity. We were hit pretty hard, but we’ve come out stronger in the end.

While we were certainly frustrated, we know that no one was more frustrated than you. Your website was down, and I humbly apologize for that. Know that throughout these six days, restoring your site and your trust has been our number one priority.

At the same time, I have never been prouder of the Posterous team. Garry led the team in the server migration, Vince worked like a mad man until he passed out on his desk, and Jackson helped the community understand the impacts of some pretty technical issues. This week has been all hands on deck, and we had some pretty great hands.

We have great plans for the future of Posterous. Our team is stronger than ever, and we are laser focused on developing better ways for you to share your content online. Thank you for believing in us, thank you for trusting us, and thank you for sticking with us through the past six days.

--Sachin Agarwal Cofounder and CEO, Posterous

This sort of crisis management is always a challenge. How do you communicate this sort of problem to customers without losing credibility later if there is a risk of lingering issues? Based on the tone of the email that you posted, as a customer, I would expect the problem solved, case closed. Certainly, it makes customers feel good that the team has it handled.

The new events, if not handled quickly, could create doubt in customer minds and the team, as the optimistic outlook may not hold. I hope they beat this soon.

Simple rule about DDoSes: there are no guarantees.

Why DDOS posterous? What's there to gain from it?

This isn't like the ones who DDOS'ed MS out of their hatred for it or the ones who blackmailed & DDOS'ed a gambling website when they refused to pay up.

someone probably has an issue w/ the blog/content of someone who is using posterous rather than posterous themselves.

That's my guess as well.

Weebly said (I think in their YC Founders at Work interview) that they get multiple DDoSes a day, aimed not at them but at sites they host. Now they have good enough systems in place to deal with it that they said they often just don't notice.

I guess when you host enough people's content, it's only a matter of time before someone wants to DoS something you're hosting.

I severely doubt this line of thought.. but they didn't make any friends with their "switch from X to us" campaign, and it only takes one bruised ego with a lack of ethics to stir up trouble.

That said, if it's like most other DDOS attacks I've seen reported, the target is probably a user of the service with Posterous merely being unlucky enough to host a particular site someone doesn't like. I vaguely recall another platform got DDOSed recently due to some Israel vs Iran type cyberwar.

Must be something that they really dislike.

Maybe, and take it with a grain of salt, some people got infuriated by their recent switch to posterous campaign. I know this may be a very remote option, but in my opinion it's very possible.

"A grain of salt" is the expression.

Thank you

Never underestimate the power of lulz when trying to understand a DDoS.

The botnet controller could be extorting websites directly... wire $5000 to my paypal account or your website will go down. Anyone from Posterous care to confirm/deny in this case?

Do you have any evidence that this has ever been done in the past?

It's not inconceivable, but it seems like a huge leap.

Google seems to provide plenty of examples. Even if it didn't, I don't understand why you see this extortion as a stretch.

It's not uncommon. There was an article a long time ago telling the story of a DDOS attack on a gambling website, and how a guy who was a philosophy major (iirc) figured out how to beat it, and then formed a company providing the same service. Forgot the url/title/etc, but it was good. Plenty more available with a search.

This is very common with online gaming (== gambling) sites, and payment systems; anything where there is obviously money being processed.

Who knew what the attackers are thinking. Not too long ago DnsMadeEasy was DDoS from Korea.

Maybe there's financial incentives?

Could this have anything to do with Posterous' risqué post claiming superiority over Tumblr?

http://www.blogherald.com/2010/06/24/posterous-slanders-tumb...

Our datacenter is experiencing heavy packet loss. We're on the line with Rackspace now.

I don't see where they stated it was a DDOS attack. Packet loss can occur due to a large number of different issues.

EDIT: They just clarified with the following: The DDoS attackers have returned and evolved their attack around our countermeasures. We expect to be back online ASAP w/ @gigenet antiddos

It's interesting that they're basically saying Rackspace's network couldn't hold up and doesn't have an effective solution for DDOS victims on their network.

Gige's DDOS protection is basically a hosted redirect and filtering system that "leverage[s] the cost of DDoS mitigation amongst a large group of businesses, giving you access to the infrastructure that would normally be out of reach financially." http://www.gigenet.com/ddos-protection.html

But if Posterous is hosted on the Rackspace cloud, why isn't there a mechanism to do this through their existing host? After all, Rackspace has an equivalent service (maybe rebranded?) that would seem to offer protection without needing to go to a 3rd party vendor. http://www.rackspace.com/managed_hosting/services/security/d...

So it's interesting that Posterous needed to go around Rackspace to get a solution.

It's tough -- if someone is hitting you at > 1gbps, that exceeds a gigabit ethernet port -- typically on shared services you only pay for that much. Any more than that and it affects all the other customers on the shared network. So yes, Rackspace has the bandwidth for it, but at the same time if you were Rackspace, would you let our DDoS take down the other hosts in the datacenter?

Gigenet / Black Lotus / Prolexic comes in and says -- we have excess bandwidth, we'll take care of it. Then they proxy clean traffic to you. Unfortunately as we are learning, it's also very expensive.

I seriously doubt this is the reason but they insulted a lot of people with their "switch to us" targeted attacks.

That was classless.

It wasn't classless, it was marketing. No one wants their competitors poaching their clients but in business you deal with it. That said, the same thought went through my mind, I wonder if it's related.

That sounds very clear and professional. Unfortunately, it doesn't always work out that way.

Most of their competitors are in the same situation they are in.

Being the first one to torpedo the other ship and not expecting someone to get upset or God forbid do something awful in return is naive.

I think there is a difference in poaching your competitor's customers and claiming they are "no good".

Can someone estimate the cost of this attack? It shouldn't be cheap to lose all these bots for a massive 1gbps attack.

I could claim with nearly all certainty that this is a result of their campaign. The attacked several communities. Not just startups, but communities with developers (wordpress, tumblr etc). Their marketing was bold; but looking back they might have been more subtle.

So this is the 2nd DDOS attack. Can DDOS attack be fended off?

2nd DDOS in a matter of a week.

By crushing the enemy.

I hope Posterous does a write-up about this on Axon Flux when this is all over. Poor fellars.

Usually it means using a hardware solution like Intruguard or going through a third party vendor that resells Intruguard and/or other proprietary anti-DDOS technology.

We've opted to use Gigenet, and they're holding the line right now quite well.

I wonder if you guys know who's doing it, and why.

Yes. It is getting DDOS'd now. I just posted a post. But can't access it.

Reply on news.ycombinator.com