Secure Access to 100 AWS Accounts

My organization evaluated Okta and I also came away worried about introducing another third party that had access to all of our services, both from a security perspective and as another point of failure.

In particular, in Okta's SWA apps they have a setting for those apps to use the same password as the user's Okta password. If there's a way they do this without storing all of their user passwords unhashed, I couldn't figure it out and the Okta reps weren't able to provide a clear answer.

Edit: I _can_ think of one potential solution: Okta initially sets user's password to a random string, when you authenticate with Okta / change your Okta password, they use the unhashed version to make admin calls to update the apps' password and never persist it. But they didn't give any indication that this was the case.

If you already have an internal directory/SSO (e.g. Active Directory) the advantage to using something like Okta (or another decent SAML app/provider) is the user experience. It makes SSO really simple (you could install the agent on a desktop), has simple directory sync and allows you to create network source rules (e.g. trusted networks) and integrates with several 2FA/MFA solutions out of the box. If you don’t already have an internal system like Active Directory it might not be worth the extra single point of failure/compromise.

I can see Amazon displacing these services over time (they have a lot of the components already) but last time I looked they weren’t there yet.

I don’t want it to seem like this is a sales pitch either. Okta has plenty of warts, odd decisions, glaringly obvious missing features and has done some bizarre 180s in terms of organizational direction.

It was a long time ago but one bug I won’t forget that was shocking: a combination of special characters in a password resulted in a sync failure for a couple of accounts. That’s bad on its own because it raises all kinds of questions about how they’re handling and sanitizing strings. But the whopper was that the errors presented to admins had the passwords in clear text. You would assume masking would be a key thing in their code and logging systems and it made me question if they even had masking at all if a user password made it all the way to the dashboard.

It should be noted that Orgs should be setup when new AWS accounts are created, as it can be tricky to perform once your have systems in production. I would even go so far as to say if you want to commit to Orgs, create a new AWS account hierarchy and move your applications and infrastructure into it.

EDIT: https://docs.aws.amazon.com/organizations/latest/userguide/o...

"Warning: We strongly recommend that you do not attach SCPs to the root of your organization without thoroughly testing the impact that the policy has on accounts. Instead, create an OU that you can move your accounts into one at a time, or at least in small numbers, to ensure that you don't inadvertently lock users out of key services."

This is really similar to what Amazon uses in-house to manage aws roles. Not sure if there’s a commercial or open-source alternative but this could be a fruitful project for someone!

That’s neat would your company consider open sourcing it if I understand correctly that you made it for your own use not to sell it anyway.

Do you have some sort of failsafe in place for developers to be able to access AWS in the event of an Okta outage? Or are the Okta user sessions long-lived enough that you'd be able to mitigate some other way if Okta was down?

Glad Segment is making good use of aws-vault and it's components! Did you consider whether adding okta support in aws-vault might have made sense? I've been thinking about doing that for a while, interested what made y'all decide to create a new one.

P.S. Any chance of a link back in the mention of aws-vault in the article? (aws-vault author here)

Hi there. I'm confused by your statement that "The integration guide recommends connecting the identity provider to each AWS account ..."

This used to be the way Okta recommended integrating to AWS, but in the guide you linked[1] they are using pretty much exactly the same approach you did - authentication to an identity account (ops account in your terminology) then assume-role into the target account.

I can't see any major differences between the standard Okta approach and the way you outlined in your blog. Could let me know if I missed something?

[1] https://support.okta.com/help/servlet/fileField?retURL=/help...

Tangential: Why Okta over some other provider? GSuite can be used as an SAML provider, can't it?

I can't answer for Segment, but our AWS account strategy is very similar, so we've found that there is still a need for a centralized KMS solution (system-to-system auth). Our teams are working on a project to wrap more tooling around Vault and make it a centralized KMS. T-Mobile has done some work on this... https://opensource.t-mobile.com/blog/posts/introducing-tvaul...

If you use aws-vault --server or something like it that mocks a metadata server, all the sdks handle rotation properly. Just need to fork to connect it to whatever generates tokens

AWS has a lot of resources that are not per region.

The GDPR line is about improving our least privilege access story. There's a lot more to GDPR than that, though

We use ECR and have all of our images in our hub account, and make them readable from the prod/stage/dev accounts using a cross account policy.

For S3, it's more complicated because of object permissions and all the insanity that comes with cross-account writing etc.

Edit for more info: No explicit promotion process for containers. Engineers build images in our CI pipeline when their pull request is approved and merged to master, which is pushed to the hub account.

You need to define “between services”. You mentioned oauth and VPC peering. One is authentication, the other is networking accessibility. There is no one solution.

Generally you have: VPC peering between accounts, Network Access Control List (NACL) for VPC port control, security groups between instances (and some AWS services which uses SG to limit port access), IAM roles to authenticate and authorized certain AWS services to do things (e.gz Lambda to read S3 bucket) - but IAM and policies (bucket policies, SQS policies) govern authentication and authorization. Finally there is also organization service which allows you to control what AWS services are allowrd in a group of AWS accounts.

Sorry, on mobile so I can’t make a prettier list. I am generally disappointed at the complexity of authentication and authorization mechanisms exists for AWS services to be really honest.

PrivateLink endpoints between accounts and VPCs if the service must remain private [1], public TLS endpoints either with or without TLS fingerprinting for services that must be public.

"You can create your own application in your VPC and configure it as an AWS PrivateLink-powered service (referred to as an endpoint service). Other AWS principals can create a connection from their VPC to your endpoint service using an interface VPC endpoint. You are the service provider, and the AWS principals that create connections to your service are service consumers."

[1] https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/endpo...

In practice, cramming all this into the same account doesn't work. Segment is following best practice here.

For example, IAM doesn't provide the granularity in resources and conditions that you'd want to effectively isolate the blast radius of developer keys. ec2:TerminateInstances didn't (doesn't?) support VPC level conditions, so being able to terminate one instance meant you could terminate all instances.

Similarly, you might want your engineering team to iam:PutUserPolicy in development, but have a much more restricted group in production which isn't possible with IAM today.

I've taken this pretty far in the past to attempt segmenting within one account, but always run into limits: https://github.com/witoff/self-service-iam

It is recommended to use multiple accounts for isolation. AWS has recently launched Organizations to make creation and management of multiple accounts simple. See https://aws.amazon.com/organizations

Tools like aws-vault and now aws-okta make securing credentials a piece of cake.

This just in: directories are pointless because you can put all files in the same one!

---

There are advantages in multiple accounts, for organization and security. Random example: adding tags to ec2 instances is an account-wide permission, for all ec2 instances. Multiple various things by 100 and accounts make sense.

It's a bad idea to use multiple env in the same AWS account.

But you have to prefix all of your resources then, i.e if you are using SQS, DynamoDB, Lambda's and it becomes a huge mess.

OP here. You _can_ do this but IAM Policy wise it is a mess and not very maintainable.

You end up needing to constantly grant and revoke access to individual resources if you went this route. Instead, it's nice to give engineers access to everything that's in their account, and not worry about IAM policies.

Lots of good information and considerations here: https://aws.amazon.com/answers/account-management/aws-multi-...

As with all of AWS (and software, and life) there is no one size fits all answer.

Their reason for multiple accounts seems to be due to API rate limiting imposed by aws? At least thats my understanding.