Hacker News Reader

Comment by etothet | Hacker News Reader

Speaking of unique names within AWS, I learned the other day that even after you delete an AWS account, you can’t reuse the root user email addresses (it’s documented, but I wasn’t aware).

Someone at my org used their main company email address for a root user om an account we just closed and a 2nd company email for our current account. We are past the time period where AWS allows for reverting the account deletion.

This now means that he isn’t allowed to use SSO via our external IdP because the email address he would use is forever attached to the deleted AWS account root user!

AWS support was rather terrible in providing help.

View on HN

AWS support seems to be struggling. I just came to help a new customer who had a rough severance with their previous key engineer. The root account password was documented, but the MFA went to his phone.

We've tried talking to everyone we can, opening tickets, chats, trying to talk to their assigned account rep, etc, no one can remove the MFA. So right now luckily they have other admin accounts, but we straight up can't access their root account. We might have to nuke the entire environment and create a new account which is VERY lame considering they have a complicated and well established AWS account.

This is why you either issue corporate phones or key dongles.

when your startup is three employees and only one technical? this person created their AWS root account, I think it's fair to assume that he's their first engineer and probably first employee

What happens when someone loses their phone?

You print the MFA QR code, and give it to an executive that locks it up in a safe or offsite storage.

In a past life, we printed the MFA QR code and the head of finance put it into a safe.

You know that QR code is just text you can read right? It's just an otpauth:// URI you can copy and paste into most password managers.

We even have these amazing things that securely share passwords or other secret data between multiple authorized users.

Seriously just scan the QR code and put it in any password manager that supports TOTP and it will start outputing codes.

That's precisely why you want it in a safe.

Yes, I am very familiar with zbarimg and qrencode. But, other people might not be, and that's why just scanning a QR code works. Not everyone has Bitwarden, 1Password, Pass, keepass, etc.... also these tools may not be approved by your security teams.

And we are talking about the root account for your production AWS account. No need to get fancy. Just print the QR code, and put it in a safe hoping you never need it.

This is why you never use personal phones for MFA to critical accounts.

Amazons assistance for account issues to organizations if an employee did anything individually is honestly horrible.

They treat it like the organization is attempting to commandeer someone else's account so all the privacy protections you expect for your own stuff is applied no matter how much you can prove it is not some other individuals account.

The best part is the billing issues that arise from that. In your example, if the previous engineer logged into that account (because they can) and racked up huge costs, assuming that account is getting billed or can be tied to your client, Amazon will demand your client pay for them, while at the same time refusing to assist in getting access to the account because it's someone else's. They hold you responsible, but unable to act in a responsible manner.

While true, the engineer would have to be a weapons grade tit to get themself in such legal trouble, and honestly deserves whatever criminal charges comes their way.

You would think they'd have a standard way to recover this, like mailing a one-time password to the account's billing address.

I named random Joe as the sole owner of "my" bank account and the bank wouldn't allow me to access "my" money!

That's not an equivalent analogy. A better analogy would be to say I had a bank account and I told my bank to call up Joe on the phone when confirmations were needed. I still have the account, but I have fallen out with Joe. I want the bank to call somebody else, but they refused to do so, even though it's my account and I'm paying the bill for it!

And we're paying extra for support!

Banks have established processes for changing signatories on business bank accounts, including in situations where a past signatory is no longer with the business.

In a nutshell: if a past signatory was a regular employee, it just takes any other signatory to remove them. If there was no other signatory, or if the past signatory was an officer, it takes a current officer (as set forth in the company's AOI or corporate minutes). Usually only the latter 2 situations of the 3 above require an in-person visit to the local branch office, and that only requires a few minutes.

Is this something where you could pay a "consulting fee" to the previous key engineer to login and remove the MFA?

I know that that's not ideal, but as a practical matter perhaps it would be easier than creating a new account, if you can get the engineer to agree to it?

I won't attempt to defend AWS here, but if any company has such incompetent IT management as to allow an individual employee to have that level of control then they kind of deserve what they get. Life is hard when you're stupid.

Is the AWS account phone number also their phone and not the business/corp phone? And you tried the dedicated lost MFA device form?

You can always use plus-addressing if your email provider supports that. AWS considers plus-addressed root emails to be unique.

Doesn’t solve the SSO issue though unless you change your login email

I don't really understand that problem, exactly. I'm not aware of any restrictions for using AWS Identity Center (SSO) with an email address that happens to be a root email for another AWS account.

I checked the documentation but I couldn't find anything to show this to be a problem other than that the practice is discouraged.

Or you don't have employees using their personal email to open corporate accounts.

Still on Amazon to clearly tell people it is this way so they can properly plan for it, but employee's email addresses really shouldn't be used for the root account.

That’s not what’s being described here. What OP described is the much more common situation where employees use a personal phone for MFA. Sure, some places issue hardware dongles and disallow authenticator apps on your personal phone, but IME most places default to just having people use their phone.

I create "job function" DLs. "Company-Region-IT Manager". Then give that DL it's own SMTP address. Then use that.

It's really nice when you have to hire someone new for the position. You add them to the DL and they're automatically in control of all those accounts.

I have no idea why more companies don't do this.

Good for them. It's amazing how pointless most security is when a 10/10 rating to some commodity communication service's support from a phisher is all it will take.

You should not have the root account be a human anyway. Make that a special account, secure the credentials and only ever use them when you screw something up really badly.

Help me understand why you would delete your AWS account if the company and email address are unchanged - I can’t see the motivation.

And on the flip side I can easily see why not allowing email addresses to be used again is a reasonable security stance, email addresses are immutable and so limiting them only to one identity seems logical.

Sounds quite frustrating for this user of course but I guess it sounds a bit silly to me.

what if you stopped using AWS for a while, then came back?

It’s not hard to imagine a case where maybe there’s 2 offices that had their own separate aws accounts and they closed one.

AWS has been around for quite a while now. It’s also not impossible to believe that there are companies out there that might have moved from aws to gcp or something, and maybe it’s time to move back.

This was a secondary AWS account in use by the company that had been in place for quite some time and that secondary account was just no longer needed. So to consolidate things down, it was deleted. Also at that time, SSO wasn't being used for anything with the company - and they were on a completely different email provider.

I'm not arguing that it was impossible to know the long term outcome here, but it doesn't mean it isn't frustrating. If you've spent any length of time working in AWS, you know that documentation can be difficult to find and parse.

I can certainly understand why the policy exists. What I think should be possible is in these situations to provide proof of ownership of the old email address so it can be released and reused somehow.

> email addresses are immutable

1. Use "admin@domain.com"

2. Let the domain registration lapse

3. Someone else registers the domain and now can't create an AWS account.

Rare but not impossible.

Sure they can. Use any other email address at domain.com to register.

Yes. There are solutions to all of these issues, but what often happens is these situations come about through the natural course of companies changing over time - different people managing accounts, different providers, etc. The happy path is easy, but the happy path is rarely the one we find ourselves walking down when we inherit a previously made decision.

> And on the flip side I can easily see why not allowing email addresses to be used again is a reasonable security stance, email addresses are immutable and so limiting them only to one identity seems logical.

If they aren't actually deleting the account in the background and so no longer have a record of that e-mail address, then they must allow re-activation of the account tied to that e-mail address using the sign-up process.

And in this case, it’s actually less secure for this one user and the account if as a workaround I’m required to create an IAM user for them (even though I can limit their use of the system).

>Help me understand why you would delete your AWS account if the company and email address are unchanged - I can’t see the motivation.

Have you ever worked in a company of any size or complexity before?

1. Multiple accounts at the same company, spun up by different teams (either different departments, regions, operating divisions, or whatever) and eventually they want to consolidate

2. Acquisitions: Company A buys Company B, an admin at Company A takes over AWS account for Company B, then they eventually work on consolidating it down to one account

In our case, this is exactly what happened. An acquisition of a company where their AWS accounts that were inherited were no longer needed.

It's such a common case, especially in tech with startups and small software companies getting gobbled up all the time I can't see how you WOULDN'T consider it a possible reason

I did something similar.

When I started, AWS was in its infancy and I was just some guy working on a special project.

Now that same account is bound into an AWS Organization.

AWS Changed. My company changed. the policies change out from under you.

I would expect the SSO configuration to map the IdP's given email into a role appropriate for the identity. What does "forever attached to the deleted AWS account root user" mean here? What is the mechanism blocking use?

I thought it worked the other way, you can have multiple accounts with the same username as long as they have different passwords

IAM users get usernames - they don’t log in with an email address. Root users log in with their email address.

That seems like a GDPR violation waiting to happen. It shouldn't be possible for them to store an email address like that forever and be in compliance.

If user foo@gmail.com violates our ToS and I suspend them, I can keep that email address forever to keep them from signing up again. They can’t just say “GDPR! You have to forget me, tee-hee!”

Any reason you won’t just use a hash?

Yep. Almost every company uses multiple vendors for things. Suppose you use a tech support helpdesk and you don't want to waste time dealing with banned ex-customers. You can't import that list of hashes into Zendesk or whatever and tell them to blocklist them.

Substitute "billing company" or "authentication provider" or "fraud detector" for "helpdesk". There are times when it's not sufficient to say "don't do business with SHA-256 hash ef61a579c907bbed674c0dbcbcf7f7af8f851538eef7b8e58c5bee0b8cfdac4a". You need to say "John Smith is banned".

This can be implemented without storing it. They could store a hash. No idea what they actually do.

A hash of a public identifier like an email is personally identifiable data.

You can always encrypt with a public key instead of hashing.

You mean 'as well as', right?

No, I mean encrypting (using a random padding like OAEP-RSA) gives an undecipherable item.

Isn’t the entire point of a cryptographically secure hash that you can’t derive the original information?

You can't derive the original better than guessing. With public identifiers you can just take a list of them and guess with those. If someone asks for your email they can hash it themselves and compare it against whatever databases.

GDPR says you are not allowed to store my data just because. If you have a good enough reason, everything is allowed.

Reply on news.ycombinator.com