The State of AWS Security
datadoghq.com
datadoghq.com
1. Avoid long lived credentials (access key/secret key pairs) and IAM users whenever possible, and instead have everything assume roles for access to resources in the same as well as other accounts. Temporary credentials from STS are the way to go.
2. Turn on IMDSv2 by default for EC2 instances. v1 is susceptible to SSRF attacks so even if you’re attaching an instance profile to an instance and doing everything else right, you’ve got a problem if somebody finds a weakness in the app running on the instance.
Also heavily recommend checking out Scott Piper’s SCP Best Practices (https://summitroute.com/blog/2020/03/25/aws_scp_best_practic...) that still hold up even if being a few years old at this point.
It sounds like really basic security - my saved credential can't go delete servers. Absolutely everyone, from aws certified people to apparently renowned developers have gotten stuck into me over this, with complaints from "I'd quit anywhere that did that" to "bad setup, proper iam never requires mfa". I'm sure there's going to be some form of blow up where long term saved credentials eventually come out as problematic, and it will blow people's minds.
MFA seems like an essential defense-in-depth measure to ensure that a compromise of the locally held IAM key is not enough on its own to compromise your AWS account.
It's really disappointed that aws-cli doesn't easily support this type of workflow, when using MFA and setting up multiple AWS accounts with cross-account roles are two things recommended as security best practices by AWS themselves.
Don't get me started on how you can only have a single U2F key attached to your root user.
Regarding root, I always create an account with a console login that can remove the root user mfa or reset a password, it becomes the recovery account and I can put its own key on it, and ideally never gets used once tested.
Say I want to load a huge amount of data, or do a ton of s3 uploads. My role assumption via MFA / TOTP lasts 4 hours.
Now say you are doing lots of different "big data" flows, how do you assume privileges when you need them?
I understand what MFA / TOTP brings to security, but security people pretend like my job is about logging into a box once-in-a-while for a couple minutes, and that is the ONLY USE CASE.
Least privilege is another thing that sounds great to a security guy but is a nightmare for development. So whenever you do anything off the beaten path (diagnose new systems deployed, try new database backends, do some data moves or loads, backup/restore), uhoh, you spend a ton of time diagnosing shortfalls and decoding error messages. The security guy DOES NOT CARE.
S3 is a perfect example. S3 should be simple. It is a nightmare of permissions on the user side, permissions on the bucket, and even worse across accounts. Tiresome. Magic numbers/dates for the "version" field policy, what is with THAT? ACLs/Policies stomping over each other.
The funny thing is, ask a security person to first enable cloudwatch and dig through the data to find issues and vulnerabilities, or ask security people to actually produce well-tuned IAM roles for what you need. Guess what? They don't want to do it. They want to sit around making checklists and sending memos.
It’s dorky, but 9pfs over virtio would work for this purpose, at least with a Linux guest.
IMDSv2 is really aiming to do two things:
1. Require a session to access credentials. You can only initiate the session with a PUT request to prevent some reverse proxy and WAF misconfigurations from becoming the front door. If your reverse proxy still supports PUT requests, IMDSv2 is going to reject anything with an X-Forwarded-For header
2. Once the session is established, it can only be used from the instance where the session began as you get an instance-specific token - so you can't take credentials and then plug them into your own client to start exfiltrating data
And this supports regular file modes and ACLs, whereas getting this right with iptables/nftables is awkward at best and requires enabling a firewall.
It's a HTTP interface so that the most software can benefit from that, whether it's a shell script using curl or something written in Java, Scala, Python, Node.js, Rust, whatever ... pretty much everything can make HTTP requests. Of course over time that HTTP interface became a target itself and when applications have SSRF issues, badness could ensue.
IMDSv2 limits the reachability in four ways to protect against SSRF. 1. The IMDS can be entirely turned off if you prefer. 2. Access requires a PUT request, our analysis showed that SSRFs that grant the ability to PUT are very rare. Most SSRF issues only grant GET because they are a URL or header escaping issue that don't give up control of the method. 3. Requests with X-Forwarded-For are denied, so if it's a misconfigured open proxy or WAF that adds this header, you get protection. 4. The request has to come from the local box because we set the IP TTL to "1" which means it isn't routable off-box. This protects against misconfigured NATs and routers.
And it's hilarious that ssh is being treated as some massive vulnerability by the security folks, and they are pushing enterprisey crap like Teleport instead, which crashes / agents go dead / doesn't work in low memory.
Yeah, SAML is, as far as I know, always headed. My copy of aws-vault is bright enough to launch against our AWS SSO provider(1), which secretly is also SAML to Okta, but I guess if your company's setup doesn't use AWS SSO then Nike must have thought parsing HTML to be The Solution™
1: https://sourcegraph.com/github.com/99designs/aws-vault/-/blo...
I say that because taking this blog post as an article, it makes one major false assumption: Credential Leakage = Compromise.
In reality, anyone who knows AWS IAM knows that you can:
- Activate/Deactivate credentials
- Time-gate policies
- IP-restrict policies
- Use roles with STS temporary credentials
- Use roles in conjunction with the externalId attribute
- Many more things I've missed out....
But that doesn't detract the fact that this blog is making a cheap-shot and not really taking into account the strengths of AWS IAM in terms of the security granularity that make it possible (with a correct, layered config) to comfortably say Credential Leakage DOES NOT equal Compromise. You just need to take a minute to RTFM and sit down and plan your IAM setup, instead of just doing it on a whim.
AWS IAM is actually one of the best bits of AWS. Its one of the areas where AWS leads in comparison to its competitors (IMHO).
No.
As per my post above, they "led to compromise" because lazy syadmins have been too inept to RTFM and make full use of IAM.
I mean, I could post my IAM credentials right here. I've even been lazy and haven't rotated them for 5 years (yes I've already slapped myself on the wrist !).
But they'll be naff all use to you because:
- My credentials are deactivated when I'm not using them.
- The STS policies they are linked to are IP-restricted
- The Role policies they are linked to are also IP-restricted
- The Role policies have minimal necessary privilege
- The Role policies can only be invoked be an assumed role
- The Roles have externalID attribute set using a UUID as value
And that's just a simple example. You can go much more layered and granular than that.
Its not rocket science, its not difficult, its just a case of the old PPPPPP (Prior Planning Prevents P* Poor Performance).
Companies generally totter along until about 50 engineers / 100 employees before hiring a single security-focused engineer. When those first few security engineers come on board, you can bet they'll have a backlog of at least several months' if not a couple years' worth of research and remediation before they get anywhere near taming the company's IAM setup. The vast majority of companies are small and probably aren't where they need to be in terms of IAM practices.
AWS adoption has become a lot more mainstream among companies. 10 years ago it was almost a secret weapon for forward-thinking (and VC-money-burning) companies that let them out-scale competitors. Unfortunately their defaults, documentation, guard rails, etc are still set up for those bleeding-edge, hyper-competent, top-0.01% companies.
If an engineer doesn't understand how to write secure software then they will continue to write un-secure software and there will be leaks and compromises. You're just externalizing risk and costs to your customers.
A public/leaked set of IAM credentials, using reasonable AWS access policies, should be considered more secure than a private set of credentials that just has unrestricted access to everything.
Indeed I'm not.
To spell it out, what I'm saying is....
People should be working on the assumption that their IAM credentials WILL be leaked and work back from there.
That leak may be accidental on Github, it may be via a hack, you might have accidentally pasted it into an email, it doesn't matter, the mechanism of leak is irrelevant.
Blast radius / Attack surface / Layered security ... whatever your choice of words, that's what you need to do and that's what AWS IAM gives you the tools to do.
As I've previously said, its really not that much effort to add the extra layers of security (e.g. you can start with easy stuff, low-hanging fruit like IP/Time constraints and then move onto more advanced stuff later).
Infact I'd argue the detractors here who have been busy downvoting me and arguing against me could have put a lot of extra security on their IAM credentials in the same timeframe. ;-)
This sounds rather interesting, mind elaborating how its done? How would they activate when you use them, but not if anyone else does?
You can quite easily monitor cloudtrail for malicious activity and use cloudwatch to notify you when someone uses your credentials from, say, an unknown IP, outside of working hours, etc.
Do you mean using roles (which aren’t credentials) or temporary credentials (which can’t be revoked)?
Or do you use other credentials to disable them (and how do you disable those)?
I’m not an AWS expert by any stretch, so genuinely interested as I wasn’t aware of any other mechanism you could be using.
Not saying they're impossible, but they are definitely difficult to figure out!
> No.
> As per my post above, they "led to compromise" because lazy syadmins have been too inept to RTFM and make full use of IAM.
You're contradicting yourself: "No, it hasn't led to compromises" but "they led to compromise because ..."
It's possible that (as you argue) AWS IAM provides strong security controls and, simultaneously, a lot of people use it in a way where credential leak _does_ represent compromise. My going-in expectation is that both of these are true. But this post isn't about whether AWS IAM is any good. Nor does the post talk about solutions -- I don't see a call to action to buy Datadog (aside from the banner in the footer that doesn't look connected to the post). So your criticism of "this is a non-problem -- everybody's just using it wrong" sounds pretty lame.
I don't follow your argument.
Perhaps let me try with simpler example.
I'll tell you my credit card PIN number is 7341.
Following your line of argument, you'll stand there screaming until you're blue in the face: "that's a secret credential leak, your card is compromised".
To which I say: "well, my card is in my wallet, and my wallet is in my pocket, so watcha' gonna' do with that PIN number chum ?"
So its the same with AWS.
I could tell you one of my access keys is AKIA3CWKZKKGZLHN and its secret access key is QOF0yG/lvqqAcklAHCDzPKRtk9D5oPnY.
But watcha gonna do with it ? Try logging in ? To what ? And even if you knew which AWS service, even with just low-hanging fruit security layers of IP range restrictions and time-gating you still won't get anywhere. Once we start adding extra layers such as roles and STS on top, then frankly you're more likely to win the jackpot on the lotto twice in a row.
If its not usable, its not compromise.
Maybe we came to this with different assumptions? My going-in assumption is that most people have not set up those extra layers that you're talking about. For them, a credential compromise is tantamount to an account compromise. Thus, a post like this is relevant to raise awareness of credential compromise (and sure, maybe a missed opportunity to talk about those other layers one could add).
Is your going-in assumption that most people _are_ using those extra layers and so the post is pointless because it _erroneously_ implies that a lot of folks might be more exposed than they think? That's not what I thought you were saying. I thought you were saying: "if a credential leak compromises your account, then you're doing it wrong". That might be true, but if there are lots of people doing it wrong, then it doesn't matter.
(I don't like your PIN example because in real life, the initial conditions set by the bank are that you know your PIN and you have your card. You have to go out of your way to expose both. By contrast, with the AWS credentials, you have to have taken the extra steps you mention to establish the extra layers that you're talking about (IP range restrictions, time gating, etc.))
Also if you push AWS keys into github (this does happen from time to time at the best organisation), github will nuke the rev. And even if you did it successfully there should be no damn keys which are usable because anything human issued should be via SSO and windowed or have MFA required set up. Everything else should be assume role based.
We did release some (hopefully) actionable guidance alongside the study[1].
Trusted Advisor is a fair point, but note that most of its security checks only come with the Business or above AWS support plan. IAM Access Analyzer is a great service but it currently supports only 6 resource types[2].
We'll look into adding both, appreciate the feedback!
[1]: https://securitylabs.datadoghq.com/articles/state-of-aws-clo...
[2]: https://docs.aws.amazon.com/IAM/latest/UserGuide/access-anal...
Another fair point is Datadog also charges for its product.
The number one cause of AWS security issues is human error or not understanding what you are doing. AWS provides a hell of a lot of tooling, documentation and guidance to manage those risks out of the box. What doesn't help is buying a vendor product first and assuming it's going to do magic unicorn farts and make everything ok. What it will do is cost you a ton of money and time to tick a box somewhere that seemed like a good idea.
Someone selling a solution to those is selling you snake oil.
All of the issues identified will be picked up by Trusted Advisor or complain loudly on the IAM dashboard. If you don't notice that or don't use it, then your funeral.
> 40 percent of organizations have at least one IAM user that has AWS Console access and does not have multi-factor authentication
Yeah, I hope your last resort, non-root account is locked in a safe place and has no multi-factor on it. Statistics are cool, but some context behind the findings is also useful.
Similar story with the active root accounts. If they exist only on paper in a safe, that's fine.
(Root user ARN is arn:aws:iam::555555555555:root versus IAM users which have ARNs like arn:aws:iam::555555555555:user/USERNAME)
Wow didn't expect that. Separating accounts is probably one of the most important things to do in AWS, for security and performance/development.
This piece is clearly "content marketing", or even "technical content marketing". The purpose is to spread awareness about Datadog and possibly lure potential clients into looking at their product offerings.
There's nothing bad with it per se; but I would take the bait if you provide some really insightful information. This piece doesn't seem to add much value.
An example of something more useful, IMHO, is this report about cloud native threats [0]. (I think it's quite stupid to ask for name, company, etc, in order to be able to download the report. Anyway...) If I'm actively into security, it provides useful information.
As an example, the report cites that "It costs $430,000 in cloud bills and resources for an attacker to generate $8,100 in cryptocurrency revenue.". This information tells me that these attackers can generate a lot of damage, and it suggests I should do something to control the spending in my cloud infra, in case one of these attacks is successful.
[0]: https://sysdig.com/resources/reports/2022-cloud-native-threa...
Would love to hear your thoughts on how we can make it "deeper" and "more accurate" in the future!
[1]: https://securitylabs.datadoghq.com/articles/state-of-aws-clo...
Audit logs are in many cases disabled by default (RDS, S3, OpenSearch, ELB).
S3 does not require TLS requests by default. ECR does not have image scanning enabled by default.
Also new accounts almost all regions enabled and a default VPC in each (and subnets, route tables, security groups, internet gateway, dhcp option set). Unused VPCs are not recommended to keep around but I suppose it makes onboarding easier.
Audit logging costs money, so I'm on the fence about that.
A default VPC is easy to disable in enterprise deployments, but for the rest of us it is necessary to do quick tests with EC2-adjacent services - I'd be in favour of it not existing until you try to launch something though.
I completely agreed though - I look at CloudConformity and so many of the warnings are for encrypted resources.
I am curious where the OP got this information. Cannot imagine AWS sharing this. Datadog or GitGuardian study mentioned in the article cannot track data for such a large fraction of AWS users either. Just curious ...