Hundreds of exposed Amazon cloud backups found leaking sensitive data | Hacker News Reader

Hundreds of exposed Amazon cloud backups found leaking sensitive data

techcrunch.com

181 pointsewood61 comments

I've been working almost exclusively in the AWS space for about 10 years now. Clients anywhere from tiny little three-person consultancies to Fortune 100. Commercial, govcloud, dozens of clients.

Never once have I ever found a use case for making public EBS snapshots.

Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Note, several of those engagements did involve multiple accounts, and the need to share / copy AMIs and/or snapshots between accounts. But never making them public.

Laziness in attempting to share data with someone in another org?

"Nope, can't access it" ...

"Nope, still can't access it"...

"My manager is harassing me to get access now"...

"Look, just make it public then change it back after I get it copied"...

This 100%. I don't do much AWS stuff - but this issue exists outside of AWS as well (inside a corp).

It's why I hate all the password rotation, double VPN stuff - people work around it too much.

All the old staff start having the tech person track their passwords or write them near the computer, all the young staff use whatever new .io domain does X super easily (but less securely) or stick things on thumb driver (no keypad) or use their personal google drive etc.

goes home completely forgetting to change it back...

The later in the day, the bigger the rationalizations.

At some point you are too stupid to be allowed to use a keyboard. And yet we have an entire culture around staying late to get stuff done that probably could have waited until tomorrow.

Yeah, that's pretty believable. Guess some places have a laxer culture and no automated scans checking for things like that.

most places

Too real...

The guy that produces that last line definitely wears a suit.

As much as I dislike the typical suit wearer, the industry as a whole would be much better if IT-people wouldn’t blame everything on them and just tell them you won’t do it. If you are a welder and you boss wants you to weld a gas tank that hasn’t been emptied you tell him that he has no idea what the thing he asks for means, explain why and wait till the gas tank is emptied and everything is checked.

I worked as a Camera guy in film and I am known to be very fast – yet I had directors who wanted things even faster. Bit there is a natural limit to how fast you can get something done without having worthless garbage as a result.

You can take certain risks, skip certain advisable steps, focus on the most essential thing etc, but below that there waits literally nothing.

In my experience taking a step back, breathing in, out, and then proceed to doing it properly is in most cases faster than following your boss into panic and ditching common sense.

It is your responsibility as a professional to say “No” or “Stop” in certain circumstances. And if they really want you to do it, write down the possible consequences of what they force you to do and make them sign that they take the full responsibility.

There is so much talk about IT security with people frowning upon silly behaviour, yet any other craft would bend over backwards before you could force them into unsafe behaviour. If engineers would build bridges like we in IT operate, we would have many collapses a day.

(Not advice to parent, just generally)

If you want to be more professional about this stuff, build up a Fuck Off Fund. Women in particular have written about fuck off funds in the context of making sure you don't have to nod along when HR says the VP who stuffed his hand down your top was "just fooling around" but - most people need that financial security any time they have to confront the boss. Save to be able to look the big boss in the face and tell them "Fuck Off". "Fuck Off" isn't the response you need when they tell you they want the database authentication disabled "just until Monday, Tuesday at the latest" that's when you want "No". But you need to know you _can_ tell them to "Fuck Off" so you actually say "No". Otherwise you may find yourself agreeing anyway.

For the youngsters out there, don't elaborate when you say "No". That is, don't mention your FOF. That weakens your position.

It is sufficient to look the manager/executive threatening you square in the eye, and state your position with deliberateness. Keep it professional, no raised voices, and be willing to walk away without hesitation if the other side gets abusive.

The really bad ones are those who tell you that if you step through that door don't bother coming back or similar, so be absolutely ready to commit. If you do the main event behind closed doors one-on-one and get that threat as you walk out, sometimes they'll come back with sugar suggesting a do-over. Generally Admiral Ackbar is right in this context, but it's your call.

The negotiation leverage that comes from the FOF is most powerfully communicated non-verbally and in a face-to-face setting, also through body language. The difference is very noticeable between those who have an FOF and those who don't, if you have enough experience. It can be faked, but it takes exceptional practice to fake. The tell starts with how fast and confident the "No" comes back.

This is the nuclear option of course. Exhaust all other avenues of reasonable negotiation first. Like an email with witnesses you pick for deliberately violating departmental policy, for example.

>If engineers would build bridges like we in IT operate, we would have many collapses a day.

Maybe a bit offtopic at this point, but whay you said reminded me of this: https://www.stilldrinking.org/programming-sucks

I really enjoyed this, thanks. Do you have any other similar ones, or tangentially related bookmarks you could share?

It reminded me a bit of James Mickens’ writings. Not quite as weird and insane as him, but close.

All of these anecdotes have something in common. That the person who is asked to do something is informed about proper procedures. That is uncommon in IT, and especially in companies where proper procedure is too costly and not worth pursuing.

It is still a communications problem. They want you to do the minimum. If I as a camera person do the real minimum the director will be pleased on set but they will get a heart attack in the editing room.

It easier to give in, take short cuts and produce garbage than explaining them under extreme stress that everything faster will result in nothing.

In my eyes it is comparable and I worked in both industries. The difference is, that as a DOP in film your name will be associated with the mess on the screen, while in IT the link is not that clear. This makes the difference.

And unless your director has a monitor (which they sometimes don’t) they cannot tell at all what you are doing. They wouldn’t even realize if you didn’t even switch the camera on.

I know that film or engineering is more

Oh no. I work with very trendy Silicon Valley folks and it’s always someone in shorts and a t-shirt. Which is to say: it’s definitely a culture issue, but it doesn’t have to be a PHB doing it. I know a lot of really smart people who are under a lot of pressure, and combined with a lack of AWS knowledge, “just make it public” is surprisingly common.

Nah. "Move fast and break things" usually has an implicit "and bypass all the security, we'll fix it later"

This seems like a bias

It is and Nassim Taleb has frame it as the expert aka the empty suit problem.

To be clear, I didn't say the bias was with the person wearing the suit.

Aren’t public EBS snapshots the underlying mechanism for public AMIs? I’ve ran into complex permissions in a golden image deploy model where the same AMI is used across multiple accounts.

There needs to be controls like S3 where you can explicitly block public data.

AWS IAM kind of sucks.

I’m convinced the last few years of ramped up concern about AWS “blast radius” is an admission by AWS professionals that NOBODY gets IAM right.

> Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Non-marketplace AMIs are built on public EBS snapshots, but that's something they should still fix. Marketplace AMIs already handle private snapshots

to synchronize backups to the onprem datacentre. https://angus.readthedocs.io/en/2014/amazon/using-ebs-snapsh...

Is it a default setting that it's public or do you have to go out of the way to do that?

>Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Maybe they're trying to reproduce functionality of docker? It would actually be extremely useful for research involving modeling/AI because you could trivially reproduce the results by bundling the exact code and data.

Edit: actually maybe I'm confusing EBS snapshots with AMIs...

The creator of the first Ubuntu distros for EC2 wrote about the dangers of public EBS snapshots 10 years ago:

https://alestic.com/2009/09/ec2-public-ebs-danger/

He just got notified by AWS a couple days ago about the public snapshot he mentioned in the article.

But at least AWS is trying to make things better here by proactively checking for public EBS snapshots and notifying people.

Public EBS snapshots are great, and thankfully a design other clouds didn't copy. I've found all kinds of stuff in there, including a 900GB Oracle backup of a publicly traded manufacturer's accounting system. It doesn't require much imagination to understand how this kind of data could be profited from, given relatively low effort

It seems unlikely a lot of people didn't already know about this, it's hard to miss even if you only spend a few days with the EC2 API, and it's also quite surprising AWS have yet to correct the design. 90% chance it is mostly a UI problem -- there are no warning labels around snapshotting in the EC2 UI

> and it's also quite surprising AWS have yet to correct the design. 90% chance it is mostly a UI problem -- there are no warning labels around snapshotting in the EC2 UI

Snapshots are private by default, you have to actively make them public (impossible if encrypted) or share them (which also requires sharing the associated keys if encrypted.)

Now, AWS hasn't wrapped the extra layer of “by default, reject any setting or policy allowing public or cross-account access unless separate additional default switches have been toggled off” thing to EBS that they have to S3. But people still expose stuff via S3, so that's hardly a panacea. At some point, one has to conclude that customers are responsible, in many cases for giving too many(or just the wrong) people admin access to their accounts.

What I mean is https://i.ibb.co/P6N35qv/Screenshot-from-2019-08-10-10-27-25... does not make it clear whatsoever that 'public' really means public. Before we get into blaming the customer, there should be a bright red warning label in that dialog. Consider that English may not be the first language of many users reaching that screen

I think they have it for some stuff elsewhere, but it doesn't seem unreasonable to make public snapshots a per-account permission that defaults to disabled, and requires an interactive UI checkbox to enable. Out of the millions of AWS accounts, public snapshots are legitimately useful to maybe 1000 tops

Well, for S3 buckets, Amazon has made it very clear when it is public. It also used to be pretty clear.

For EBS - nothing is public by default, so customers have to willingly decide to click buttons to make it public.

By default, if I create a snapshot, it is NOT public...

How do you scour for EBS snapshots and open browsable S3 buckets?

All public EBS snapshots appear in the EC2 > Snapshots section in the AWS UI. Toggle the dropdown in the top right of the table to "Public" and you'll see them. Sort by size and you'll get some interesting looking ones at the top.

It reminds me a bit of old time cdroms.

For EBS, step 1 is reading the docs, step 2 is cutpasting a documentation example.

For S3 I'm not sure how people are building their lists. AFAIK the API provides no enumeration. So this is possibly something coming from web crawl data (e.g. common crawl)

Perhaps something like this?

https://github.com/eth0izzle/bucket-stream

Oh god, how much I hate articles that don‘t list their sources. Where are the slides from?

The talk description is here: https://www.defcon.org/html/defcon-27/dc-27-speakers.html#Mo...

AWS Trusted Advisor has warned of this since 2017:

https://aws.amazon.com/about-aws/whats-new/2017/06/aws-trust...

I just checked an EC2 console and I can see 19,356 snapshots created by other users.

I am so confused.

It would be trivial to make finding a snapshot require knowing a unique ID like an AMI.

And, why do I need to be able to search for 1000s of customers' public snapshots in the EC2 console? What conceivable purpose does that serve except being a giant opsec fail?

looks like some bigger images were of wiki[pedia|media] backups so I guess easy-to-find/use public data?

That’s per region, too.

It’s still true that most security issues are caused by human ineptitude, not clever vulnerability-hunting or burning sophisticated zero-days.

I would replace "human ineptitude" with "flawed system design that makes it very easy to make very bad mistakes"

"a common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools" - Douglas Adams

But is the flawed system design here the automated system at AWS or the human-in-the-loop systems by which companies are providing admin access to IT resources, including AWS accounts?

Probably both! But I would argue that below a certain size provisioning things by hand probably makes sense. A UI that makes it too easy to make a private thing public is never ok.

Sure, but I don't think the current EC2 UI makes it too easy (and the S3 UI could only make it harder by not making public and cross account access possible at all.)

That's a valid way to categorize all memory errors in C.

And I think we should absolutely hold that against C as a development language.

Once I mentioned on a mailing list Chrome’s reaction to mouse driver bug on my computer, it would buffer JavaScript events, and process it even for pages on a different domain.

Later I told the EFF I had a suspicion that iOS didn’t rate limit input events to the lock screen, independently a research found out about it a month later.

Even if there are zero days, I don’t think finding them is a particularly noteworthy or rewarding task.

Gotta appreciate the hijack of the back button on techcrunch. Bounce rate too big?

FWIW same thing is possible with RDS db snapshots and dbcluster snapshots.

I had a simple glance in the console and there are like 20,000 exposed ebs snapshots - available for anyone to copy and examine - I think that's only for a single region too - switch regions to see more.

Amazon should make an emergency decision to make all these private.

Sure it will break stuff but I'd be disappointed if Amazon left what is in effect a security hole open for the sake of backwards compatibility.

They should also give me a single click link when I sign in to show me all of my public ebs snapshots and throw it hard in my face when I sign in to the console so I simply cannot avoid seeing them all.

I have multiple AWS accounts and I just signed in to try to see if I have any public EBS snapshots and then I realised I would need to search every single region in every single account and then select every snapshot one by one to find out. That's a huge problem. I need a single click to show me every exposed snapshot across every region in my account.

UPDATE:

I can't say for sure if this is 100% right but I think if you sign in to your AWS account, then click on each of these links, you will find if you have public snapshots.

Maybe someone else could confirm if this is correct?

https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=...

https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=...

https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=...

https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=...

https://ca-central-1.console.aws.amazon.com/ec2/v2/home?regi...

https://eu-central-1.console.aws.amazon.com/ec2/v2/home?regi...

https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-west-2.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-west-3.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-north-1.console.aws.amazon.com/ec2/v2/home?region...

https://ap-east-1.console.aws.amazon.com/ec2/v2/home?region=...