Ask HN: Buffer got hacked - does anyone know details? | Hacker News Reader

Ask HN: Buffer got hacked - does anyone know details?

116 pointsandreasklinger80 comments

It seems like @buffer got hacked.

https://twitter.com/search?q=%40buffer&src=typd&f=realtime

The guys seem to be on it.

Last information: They confirmed they got hacked turned of publishing and they are investigating the issue.

I greatly apologize for the mess. Thanks for posting this! We're looking into this right now.

- All FB posts via Buffer are temporarily deleted. They will reappear again once we've enabled the FB app again (which we'll do once we're sure we're not compromised anymore).

- All Tweets are also stopped from sending.

We'll keep you updated from the @Buffer twitter and FB account.

- Leo

Do you know what has been compromised?

How do you store the Buffer passwords? SHA1(password+salt)?

Public Service Announcement: "SHA1(password+salt)" is an extremely unsafe way to store passwords. Use PBKDF2, bcrypt or scrypt.

Edit (T+7 minutes): Rewritten to not be a jerk.

I know that PBKDF1 is deprecated, but using SHA1 is an upgrade to PBKDF1 which was once the standard. I'm not sure if "extremely unsafe" is the right level of worry. You can still do iterations to make the hashing slower, if for some reason you don't do the right thing and use the methods you listed.

Is there a good way to 2 way encrypt something in something like rails where the app needs to use the decrypted token. If you have app logic doing decryption then someone can just look at the source to figure out how to do this. Is the only good solution to use a compiled language?

Compiled programs are as easy to read the source for (at least the relevant bits) as interpreted ones. Still, if you encrypt it and someone gains access only to the database, they wouldn't be able to use the tokens.

Every little bit helps.

Extremely unsafe? Care to explain why?

Hashes like SHA1 and MD5 are fast. This makes them great for things like verifying file contents... You can hash a lot of data and get your answer very quickly.

This is exactly why they are not particularly well suited to passwords. A brute-force attack (since you're salting your passwords I'm sure!) against them isn't a huge undertaking. Just with the equipment in the computer I'm on right now, I'm looking at generating about 11.5b MD5 hashes per second, or 3.1b SHA1 hashes per second.

At eight characters a-zA-Z0-9, I'm looking at about 5 and a half minutes to brute force every combination with MD5. Under a day for SHA1.

Hashes like bcrypt and scrypt, on the other hand, are designed to be slow. Their complexity factors actually provide means to slow the hashes down even further as hardware becomes faster. Instead of 11.5b/second, you can increase the complexity until you're only able to generate one hash per second... Now it takes you over three and a half centuries to hash what might take one second with MD5.

Even ignoring the possibilities of MD5/SHA1 being 'broken', they're simply too fast to be considered for hashing passwords.

(Estimates of GPU hashing speed taken from http://golubev.com/gpuest.htm).

But it appears that almost all user passwords (99.8%) appear in the top 10,000 list [1]. So even a brute-force attack on a slow hash like bcrypt is pretty cheap in the vast majority of cases. So switching from md5 to bcrypt doesn't improve your security much.

[1] http://xato.net/passwords/more-top-worst-passwords/

According to that one guy with that one list. I acknowledge that the top N passwords are X% of all user passwords. I sincerely question the 99.8% figure. The problem with doing studies like this is while we have some fairly big password dumps we still do not have the universe. Furthermore, there are some number of un-cracked passwords in the dumps we have. Further complicating the situation are password policies which may reject common passwords. It has been many years since we learned that "password" is the most common password and is commonly disallowed.

Therefore 1) it is not useless to not increase your storage security even if Y% of your users use bad passwords as you are protecting 100%-Y% of you users. 2) Y% is probably not 99.8% for you, and if you are worried about it you can take steps to mitigate the problem.

ps. He is ignoring punctuation which is an important detail for actually doing the cracking.

pps. I appreciate the sentiment (users choose shitty passwords) but not the conclusion (so don't bother storing them well). The proper conclusion is use scrypt/bcrypt and increase the work factor. You can take reasonable steps to protect your users and you should.

Yes the figure of 99.8% does seem a little high. After a bit more research it seems Mr Burnett himself can see 'a few flaws' with that figure [1].

Just to clarify: my original point wasn't that you should continue using md5. Rather, it was that bcrypt doesn't improve your security much. Given the problems with the 99.8% figure, it would be better to say, "the extra security that bcrypt provides might be less than you expect".

[1] https://xato.net/passwords/how-i-collect-passwords/

Read this:

How To Safely Store A Password http://codahale.com/how-to-safely-store-a-password/

@julien_c - broken nearly immediately.

We're looking into this right now.

How'd they get into your system? Was it a Buffer Overflow?

Good work handling this; I'm glad to know that I didn't need to revoke access because you guys were already on top of it.

I'd revoke access to the app on Twitter also, and reauthorize it back when its fixed. http://mypermissions.org/

I did that, but the FB app was already taken care of fortunately.

Best of luck. I, and hopefully everyone else, will be here, ready to buffer more content, when you get it sorted out.

Are you affiliated with Buffer App? Your profile is not that helpful …

Leo is co-founder of Buffer.

Thanks!

Just got this email:

Hi there,

I wanted to get in touch to apologize for the awful experience we've caused many of you on your weekend. Buffer was hacked around 1 hour ago, and many of you may have experienced spam posts sent from you via Buffer. I can only understand how angry and disappointed you must be right now.

Not everyone who has signed up for Buffer has been affected, but you may want to check on your accounts. We're working hard to fix this problem right now and we're expecting to have everything back to normal shortly.

We're posting continual updates on the Buffer Facebook page and the Buffer Twitter page to keep you in the loop on everything.

The best steps for you to take right now and important information for you:

Remove any postings from your Facebook page or Twitter page that look like spam Keep an eye on Buffer's Twitter page and Facebook page Your Buffer passwords are not affected No billing or payment information was affected or exposed All Facebook posts sent via Buffer have been temporarily hidden and will reappear once we've resolved this situation I am incredibly sorry this has happened and affected you and your company. We're working around the clock right now to get this resolved and we'll continue to post updates on Facebook and Twitter.

If you have any questions at all, please respond to this email. Understandably, a lot of people have emailed us, so we might take a short while to get back to everyone, but we will respond to every single email.

- Joel and the Buffer team

Hmm, I was affected (my twitter account sent a spam post) but I didn't receive an email. I only found out about it via HN. :/

Same here, I found out from text messages from friends this morning saying my Facebook had been hacked, but no e-mail from them and the "Delete Account" functionality on their website does nothing.

Sending large amount of emails takes time, I guess you'll receive it later.

Outstanding email. Being hacked sucks but Buffer is handling it very well, at least communication-wise.

They're asking people to revoke Buffer as an app: https://www.facebook.com/bufferapp/posts/566007046805080

Brutal. Once you revoke it, there's little to no chance that you're going to re-invoke it.

I've seen stuff like this get ugly in the past [1] -- I really like and respect the Buffer founders and hope they can recover from this.

[1]: http://en.wikipedia.org/wiki/Gnolia

Wow that's terrible, I guess it means they stored their access tokens with no encryption.

Personally feeling slightly smug now, I actually applied for a job there not too long ago and got a half hearted rejection without any feedback (or the courage to actually reject me). I have direct experience in doing security for a Facebook SPMD - we never got successfully hacked, and yet we ran scenarios like this in our mind all the time.

Maybe I am misunderstanding, but how is it possible to encrypt the access token? Doesn't the server need it in plaintext to make requests?

If the web server is making API calls, encryption could still protect you from some SQL injection attacks, even if the decryption key for all tokens was in the web app code.

With a system of this scale, the web server will probably be making no API calls, as everything might be sent via some distributed system.

There are, anyway, many solutions to store the keys, a key vault that is separate from both database and application/web servers would be ideal.

There's not much to do if an attacker gains control of every machine, so the general idea is to keep every attack vector as isolated as possible.

Remember encryption is not the same as hashing :) With encryption you can reverse the process, as long as you have the encryption key(s). With hashing, you can't.

I realize, but whatever server is making the API calls needs unrestricted access to the key and the encrypted tokens. So if server is compromised, the access tokens are necessarily compromised too, aren't they?

> I guess it means they stored their access tokens with no encryption

Or the encryption key was on a system the hacker may have accessed.

Yeah, same thing. I remember a large retailer I did some pentesting for a long time ago, they kept the encryption key for credit card numbers in the same table as the card numbers. Doh.

people love buffer

we'll re-invoke it

Yup, I'll keep using it. By advising users to revoke access they're clearly putting user safety ahead of their corporate metrics. That shows that they have their priorities straight and are worth working with.

Unfortunately not everyone is as forgiving.

That said, HN pathologically overestimates the impact of mistakes that companies make. It probably won't be as bad as people say here.

True. In the end, they'll lose a LOT of free users who won't bother to reactivate Buffer. A good % of paid users might stay. Hell, that might even be a net benefit - less resources to devote to supporting the free users.

I will certainly use Buffer App again. For the same reason I used Buffer App before this incident. That might be easier because my accounts were not affected. In addition, the Buffer App team seems to be working hard to resolve the incident. Security issues are unpleasant but unavoidable.

Sorry about this everyone, I know we messed up a lot of peoples' weekends.

We're working pretty hard to get to the bottom of everything.

Let me know if you have questions, I'll try and answer them all.

Hi Joel, any thing you would recommend other devs who're connecting with facebook or any other social media API to look into? maybe you can share what you guys have learned reg: security and how to do it better from this

The best thing we've learned here is to enable a setting Facebook has called "Require AppSecret Proof for Server API calls". They actually have a lot of great security features which we've not been making use of.

I think considering the gravity of the situation, you and your team has handled this nightmare pretty well so far. Any of us who write software could end up in a similar situation - be it caused by a small configuration mistake, a slightly out of date 3rd party library, or a skilled, determined attacker.

When I see a local credit union site written in VBScript get hacked, I don't worry too much because it was probably riddled with SQL-injection bugs that any script kiddie can exploit. However, I know that you guys take data security seriously (you and I Skyped a while ago) and yet this happened. That's what worries devs like me the most.

I think it would be very helpful to us all (including many of your tech-savvy users) if you would write a detailed blog entry describing what actually happened and how. I believe you're encrypting tokens now but how did they get exposed in the first place? Was it a framework issue? Unhandled exceptions? Wrong chmod on a log file? New employee hooking up trojan'ed PC to your internal network? Here's hoping you feel comfortable in sharing.

Buffer is handling this exceptionally well; I'm impressed and it isn't easy. We've caught bugs that lead to this sort of exploitation in the past, so if anyone from Buffer is reading this: we're happy to help you out and offer a few free months of website security scanning while you figure out exactly what happened. Just email me at borski@tinfoilsecurity.com

I'm sorry for the Buffer team, it's not going to be a fun Saturday. The Buffer team is superb, both on product and customer service, so I'm sure things will be sorted out pretty quickly.

I really feel for the Buffer team. They make an awesome product and it's hard to prevent things like this, especially when social media doesn't generally have great security right now.

Though I don't want to be crass, I can't help but mention that I'm working on a project to help minimize the damage from these sorts of things. Would be interested in any feedback. (http://socialsecurity.io)

Can anybody explain weight loss spam to me? Is it just to gain credit card details, advertising clicks?

>Accounting for how much spam actually reaches the inbox, we estimate that only about 1 in 25,000 people needs to succumb to the temptation to make a grey-market purchase to make it protable for spammers to inundate everyone with advertisements at current levels.

http://www.davidreiley.com/papers/SpamEconomics.pdf

Thanks for the link, that's a fascinating paper.

One thing that drives me nuts about that paper format (a LaTEX template, I guess) is that it doesn't have a date. And this happens a lot. If the topic was an abstract mathematical concept, evergreen and timeless, then the date isn't so important. But a paper about spam that is vague on the date? I don't understand. That field is always changing, and I presume the economics are changing, too.

Published in February 2012, revised in June 2012, according to this:

http://www.davidreiley.com/papers/SpamEconomics.html

This is rather typical, because when you prepare a paper you don't yet know the publication date, so your submitted/final version can't include any date, but the published version tends to be behind a paywall.

http://www.slate.com/articles/business/moneybox/2013/07/how_...

All our recent posts on social media (facebook, twitter) via buffer seem to have been automatically deleted. That's a lot of engaging posts, gone in a poof!

So it seems that they deleted all old posts without checking source?

No, they just temporarily disabled their FB app. Once they re-enable, the posts should reappear...

How can the posts reappear? Aren't they just pushed and then stored on the FB database?

"We've temporarily disabled our app for Facebook. All previous Buffer posts on FB are hidden for now. We'll update again soon. - Carolyn"

Source: https://twitter.com/buffer/status/394182270522646529

I posted something on Facebook too it seemed, removed it instant. But yeah, kind of makes you think twice of what services to use

shit happens

buffer rocks and is handling this in an A+ way

Yes, they do, still makes you think twice don't it?

No. The ones that make me think twice either A) don't disclose, or B) handle the fallout terribly.

Buffer is handling this well, and will continue to get my business.

I don't think you can find a service that makes a promise that they will never got hacked or be down. Common, look at what they've done to the situation.

Here's the full, official response from Buffer: http://open.bufferapp.com/buffer-has-been-hacked-here-is-wha...

As bad publicity is publicity too; what is Buffer if I may ask? First time I heard of it.

Buffer is a service where you can link your Twitter/Facebook/etc accounts and share them at timed intervals. It "buffers" your updates for you so you can release them on a specific timeline.

Buffer is a great app, particularly for Twitter. I use it regularly across all devices. I don't use it for Facebook though. I haven't observed anything untoward on my Twitter account. I am tempted to temporarily revoke app permissions on Twitter though.

Before rushing to disable, do note:

All updates have been paused.

Passwords and payment information have NOT been compromised. --

It would be sorted out in a bit.

Kudos to Joel, Leo and the team for how they're handling it so far. The speed with which Joel had a comprehensive email update out to every user was impressive. I hope they resolve it soon.

How did they store the Facebook Graph tokens in the db? It's akin to a password so here's hoping they used 2-way encryption.

Providers should create anti-spam system, so when Clients get hacked they don't receive tons of spam on users' walls

My twitter account seems unaffected - I've disabled the app in setting for now though.

(I will be re-enabling ASAP - I use this app a lot!)

There were no spam posts no the 5 accounts I've enabled so doesn't seem to effect everyone.

@fredwilson account also got hacked. batch.me anyone (product plug ;)

In my opinion its poor form in my opinion to be plugging your product here.

Agreed. Unf. can't delete what I posted - didn't mean the plug in a malicious way. Buffer is a far superior and loved product (I/friends use it). batch.me is a small weekend experiment a few friends and I put together about a year ago. PS: Buffer team is doing an incredible job trying to put out this fire - so props.

They have since disabled the Facebook app.

Glad I used desktop twitter management apps like this one http://bit.ly/auto_app

Buffer is not only for Twitter. Or what you want is a bunch of apps for every single media you are in.

my point was being a centralized app like buffer , it will affect all users. But using an individual desktop client will prevent being hacked centrally.

Reply on news.ycombinator.com