Amazon ELBs are vulnerable to Heartbleed | Hacker News Reader

Amazon ELBs are vulnerable to Heartbleed

forums.aws.amazon.com

140 pointslox49 comments

amazon's response on this has been absolutely disgusting. we pay for their enterprise support, and that team is completely unaware. last night they advised that if i wanted this fixed in a day i should consider terminating my own ssl. i set that up, and as soon as i was about to cut over, i noticed that our elbs were fixed.

as of now, support is unaware there is a fix being rolled out.

i would have been better served not speaking to them, let alone paying for aws support.

I've had really mixed experiences with support from them, ranging from this to awesome. Mostly it's been ok - except for their more legacy products. Sucks you wasted your time, but I'm glad it's fixed!

While stock service support is good (EC2, S3, etc), we've had terrible support with Elastic Transcoder. 1 week+ on an issue that completely breaks HLS video encoding. We switched to encoding.com because I get to talk to a human being, over the phone, and aren't put through a support forum.

We've had the exact opposite experience: After reaching out to Amazon our ELBs were prioritized for the fix and patched with no work needed on our part within <8 hours. Our rep has been in constant contact the whole time.

did you contact your account rep or go through support? if the former, that makes sense. ours is traveling..

We filed a ticket through support and also contacted our rep, referencing the ticket.

It looks like they are actually fixed, as of 1 hour ago:

https://twitter.com/lox/status/453443517017100288 http://filippo.io/Heartbleed/#99designs.com

Our servers in us-west-1 are still showing as vulnerable sometimes. Even with that 99 designs link, if I refresh sometimes if passes and sometimes it fails. Possibly the patch is still being rolled out?

I think they started rolling it out into eu-west too. Our hosts return "vulnerable" 50% of the time now.

You could try creating a new load balancer, attaching your instances to the new ELB, and destroying the at-risk ELB(s).

Not in eu it seems.

Beware that that tool is often returning that your domain is not vulnerable when really it is. Appears to be a load issue. I'm not confident that any ELBs are patched yet.

Seems like CloudFront is still vulnerable: http://filippo.io/Heartbleed/#d8u1nmttd4enu.cloudfront.net:4...

According cloudflare blog: Today a new vulnerability was announced in OpenSSL 1.0.1 that allows an attacker to reveal up to 64kB of memory to a connected client or server (CVE-2014-0160). We fixed this vulnerability last week before it was made public. All sites that use CloudFlare for SSL have received this fix and are automatically protected.

[1]http://blog.cloudflare.com/staying-ahead-of-openssl-vulnerab...

I think you're conflating CloudFlare, the company, and CloudFront, the AWS CDN service.

In a same manner CloudFlare had it before the disclosure, OpenSSL team should've contacted major GNU distro (Debian, Fedora, Arch) packagers privately and do the announcement as new releases hit the repos (i.e. not having a 4-8 hour window, given the bug's pretty much critical).

I was under the impression that they did in fact contact package maintainers in addition to companies like CloudFlare.

Nope; package maintainers said they didn't get notified, and OpenSSL explicitly has no notification mechanism for such things. CF found out because the private entities which found the bug warned them a priori with a request to not disclose it to anyone else. See also: https://news.ycombinator.com/item?id=7549986

Here is the official update: http://aws.amazon.com/security/security-bulletins/aws-servic...

Werner posted this:

https://aws.amazon.com/security/security-bulletins/heartblee...

Wonder where this is even linked from?

Apparently there is a Security Bulletins page, but I wasn't aware of it until you posted this link. https://aws.amazon.com/security/security-bulletins/

Just used Zapier to set up a RSS-to-email trigger to get notified about things like this in the future, although Amazon really should be sending them out automatically to customers.

Great idea, I just setup an RSS-to-slack trigger for future notifications: http://zpr.io/HSkn

Should anyone else find it useful, here's what I'm now using: http://zpr.io/HS5f

It detects new AWS security bulletin items and notifies you via Google Hangout.

Great!

./heartbleeder zapier.com

VULNERABLE - zapier.com:443 has the heartbeat extension enabled and is vulnerable to CVE-2014-0160

AWS's ELB (which we use) were vulnerable, we'll be replacing certificates ASAP. We (and most the rest of the internet using ELB) seem to be in the clear now:

    ./heartbleeder zapier.com
    SECURE - zapier.com:443 has the heartbeat extension enabled, but timed out after a malformed heartbeat (this likely means that it is not vulnerable)

When did you run your check? Do you have a recent binary of heartbleeder?

That'd be great, except the last thing in the RSS feed at feed://aws.amazon.com/rss/security.rss is almost a year old ಠ_ಠ

How CloudFlare got patched last week and AWS not?

I am also interested to know why CloudFlare is ahead of those major Linux distributions on this.

Simple. Someone alerted them ahead of time since they terminate a lot of SSL connections. Testing with distros is a lot harder.

CloudFlare will not terminate SSL for free users - only for the paid plans. I will be also surprised if they really terminate more SSL connections than AWS ELB & CloudFront. I am not saying CloudFlare did anything wrong - they are doing great this time. I am wondering if other parties like AWS received the disclosure at the same time, what they are waiting? CloudFlare fixed the issue last week, not yesterday.

Heroku is also still vulnerable, which makes me slightly uncomfortable.

Me too. And I feel they won't be the fastest here ... Certainly a downside of Heroku, where you cannot just re-compile and fix things yourself.

There are new tools[1] being built around Docker that make it viable to easily move your apps out of Heroku and onto any system running Docker including DigitalOcean, GCE, AWS, etc.

When you combine Docker, buildpacks, and CoreOS[2], you get a scalable and flexible platform that you can run anywhere. It has taken people a long time to combine the simplicity of Heroku with the flexibility of bare metal, but the open source guys have finally put all the building blocks together.

[1] https://github.com/CenturyLinkLabs/building/

[2] http://www.centurylinklabs.com/building-your-first-app-on-co...

I've been experimenting with Cloud66 the past few days to accomplish much of the same thing, and am pretty happy with the result.

Another tool that does just this would be Deis[1]. We specifically combine CoreOS (in heavy development[2]), Docker, Heroku Buildpacks, and also Dockerfile deployments[3], too!

[1]: http://deis.io/

[2]: https://github.com/opdemand/deis/issues/530

[3]: http://docs.deis.io/en/latest/developer/dockerfile/#deploy-u...

Seems they were pretty fast: https://status.heroku.com/incidents/606#update-1970

We've posted a full update on the blog: https://blog.heroku.com/archives/2014/4/8/openssl_heartbleed...

We have over 200 ELBs, and therefore over 200 SSL certificates to go and manually replace. Time for gin.

time to automate - http://docs.aws.amazon.com/ElasticLoadBalancing/latest/APIRe... :-)

Oh, we automate that end of things, our cloud is all chef managed... but getting 200 certs re-issued by a dozen different issuers with new private keys is, how you say, tedious.

Yep. Still time for $drinkOfChoice. I threw my whole todo list for today out the window this morning.

I have 3 ELBs. 1 seems fixed, and the other 2 still vulnerable...get your shit together Amazon!

I manually created a new ELB which seemed to have been fixed and then updated DNS.

Reply on news.ycombinator.com