Amazon EC2 Enhances Defense in Depth with Default IMDSv2 | Hacker News Reader

Amazon EC2 Enhances Defense in Depth with Default IMDSv2

infoq.com

39 pointsrbanffy56 comments

The IMDS is useful, and these changes definitely make it safer, but it is also inherently risky. I wrote a blog post[0] about the risks it introduces. One of them being that any process on your VM instances can hit the IMDS and get credentials, allowing it to perform any action that the assigned role can perform. A simple role that allows apps to write logs to a bucket could be used for exfiltration if the policy is not fine grained enough.

[0] https://cedwards.xyz/instance-metadata-service-risks/

The problem is that the FAANGS run single-process servers at a scale of thousands of VMs, so in their mind 1 VM == 1 App role. They have no concept of sharing a single VM for dozens of apps.

Well, they're kind of right. Physical servers I get, you can't really do 1:1 without wasting a ton of resources. But VMs can be resized quickly.

The problem is that many companies have very rigid IT departments so even though in theory it can be done somewhat easily, in some places it can be a months-long process to actually get it done.

You can’t resize a VM below 1 vCPU. Actually in most clouds it’s 2 vCPUs minimum in practice.

I have a government customer running 150 web apps in about 20 vCPUs. If you naively multiply 150 apps by the minimum scale for HA, non-prod, and the 2 vCPU minimum for cloud hosts you end up with 450-600 vCPUs! That’s expensive even for governments.

I don't understand why you'd want to use VMs for something like this when containers exist and you have tools like ECS and Kubernetes. You can serve the same amount of web applications on the same amount of vCPUs and memory footprint, give or take a little bit. As a bonus, if you used AWS ECS, you also get to use things like IMDS (per container/task), Fargate, high availability and autoscaling tools.

Windows.

While it's technically possible to run something like an old ASP.NET Web Forms app in a container, it'll be painful.

> Windows.

You should have put that in your first comment, I've removed my downvote!

We are really enjoying using Azure Functions to serve web traffic these days. Probably not in exactly the way Microsoft would prefer us to, but it certainly works for us. All our business really requires is simple web forms, some session state and basic get/post request handling. Our users won't cry if they see a page transition.

Az functions might not be the most optimal in terms of per request cost, but for the volume of traffic we are pushing it doesn't scale into the equation.

Operationally, they're basically magic. TLS, auth, etc. all handled. GitHub deploys with zero drama. Whatever margin we sacrifice at the request billing is more than compensated by how well our engineers are sleeping at night.

A. EKS and Docker brings its own complexity

B. Enterprises run a ton of commercial off the shelf software hosted on VMs

C. Ever heard of Windows? It’s still big in the enterprise

While EKS can be a complicated beast to manage, Fargate (ECS) has been a pleasure to deploy and manage, and with ECS Exec you can jump into the container as if it were a VM. The most difficult part would be getting someone to throw their app into the container to begin with.

Note: you can run Windows containers in both ECS and Kubernetes! I've never heard of anyone doing it but it's possible.

> You can’t resize a VM below 1 vCPU.

There's also shared vs dedicated and burstable options if you need to go lower. Which clouds don't (in practice) offer 1 vCPU?

"1 vCPU" is arbitrary unit, you can run as many vCPUs on physical CPUs as you want. AWS Lambda gives you ~1/14th CPU as a "vCPU" on minimum (128MB config), t3.nano has "5% baseline performance/vCPU". So VMs do not need to be expensive.

The irony is that CPU is a compressible resource, and the cloud provider is almost assuredly over-subscribing customer loads.

GovCloud may be different for security purposes, not sure.

> They have no concept of sharing a single VM for dozens of apps

They do, it's called a container or a function, which have their own IAM/auth/role.

In the context of AWS, both containers (Fargate) and functions (Lambda) run on Firecracker, meaning separate VM for each container/function. No shared VMs there.

Separate micro-VM which is just there as a security boundary and nothing else, so I'm not sure it's comparable to an EC2.

That's what EKS is for...

Yes and that’s a simple solution…

IMDS is just one option, you can still pass IAM roles to applications in other ways.

This comment is so odd it's actually doing my head in.

Google invented Kubernetes, Apple runs Kubernetes and previously was heavily into Mesos, likewise for Netflix. And I've seen a lot of Engineering tech blogs and worked on AWS for a decade and never have I ever seen anyone spinning up thousands of micro-VMs in that way.

Not only are there all sorts of account limits you would immediately hit and have to negotiate but the networking would be problematic to say the least.

Meanwhile you can spin up ECS, EKS, Fargate which allow you to run thousands of containerised apps without any headaches at all. Plus they support you having unique IAM roles for each container.

Kubernetes? You mean the container orchestration system where they forgot to add Multi-tenancy? And no, Namespaces are not Multi-tenancy...

https://www.vcluster.com/

What kind of multi-tenancy are you trying to solve that Kubernetes namespaces cannot?

Kubernetes has a lot of limitations from a multi tenancy perspective.

It's functional, but I think it's not as polished as the rest of Kubernetes which is why Kubernetes has a multi tenancy SIG that spawned the hierarchical namespace controller (https://github.com/kubernetes-sigs/hierarchical-namespaces) and virtual clusters (https://github.com/kubernetes-sigs/cluster-api-provider-nest...)

The one that the community recognizes never been there in the first place...

"Multi-tenancy in Kubernetes: implementation and optimization" - https://www.cncf.io/blog/2022/11/09/multi-tenancy-in-kuberne...

"...As we can see from the above discussion, the sharing of Kubernetes clusters is not an easy task; multi-tenancy is not a built-in feature of Kubernetes and can only be implemented with other projects’ support in tenant isolation on both control and data planes. This has resulted in considerable learning and adaptation costs for the entire solution. As a result, we are seeing an increasing number of users adopt multi-cluster solutions instead..."

You do not run EKS “with no headaches” compared to just running a service on EC2.

While ECS/Fargate is easier, it’s still not headache free and you’re not taking into account how many large enterprises run lots of commercial off the shelf software on VMs.

Just wanted to say, from the wording of your comment and the context, I didn't expect to learn as much as I did from your article. Very insightful with lots of good references. Top job. Thought I'd mention for others that think they know IMDS fairly well

Thank you, I'm glad it was useful.

You can use iptables to allow access only from the root user. That is what we do. I feel like that help to a large degree, assuming you properly run the applications under separate users.

I mention this in my blog post. You can do this, but then you're forced to run all apps that require the IMDS under that specific user. It's better than having no restrictions at all, but having a user shared between multiple apps is also problematic in itself.

Not really. You can have a daemon, that fetches from IMDS just what the particular user needs and stores it in a way accessible to just that user.

Well yes, you can do that, but it won't be compatible with the cloud SDKs so you'd have to write your own code to fetch that stuff, as well as implement your own token renewal since IMDS credentials expire fairly quickly. This is easier said than done on a brownfield estate.

> Well yes, you can do that, but it won't be compatible with the cloud SDKs

I am confused why you just could not use the AWS_WEB_IDENTITY_TOKEN_FILE. You could just produce that file for every user, refresh it before expiration and set the environment variable for each user. I believe that should work with the default SDK just fine, no?

I mean, sure, it is not exactly zero effort, but still seems fairly doable...

I have only had a brief look but seems you would also need to specify the role ARN at least, so it would not be as simple as IMDS-sourced credentials. This solution is also AWS-specific, not sure how it would be solved for Google Cloud. It doesn't solve the issue of getting regular metadata without sidestepping the SDKs.

Not really, afaik you can just route requests transparently to your intercepting proxy to do access control. See http://www.daemonology.net/blog/2020-01-27-Announcing-imds-f... as an example.

I also mention imds-filterd in the post I linked... it has no port to Linux because it uses a FreeBSD feature.

> Furthermore, AWS and Google Cloud’s metadata services only support one role/service account per instance. If you have multiple services that require different permissions on an instance, they’ll have to share, which is poor security practice. A service may get access to something it doesn’t need and audit logs will be less useful. There’s no such limitation with static file-sourced credentials.

Bit of a head-scratcher. If you run multiple services on a VM, why not issue each of them identities of their own, rather than relying on the IMDS identity? If you need separate identities on the same machine for your security posture, then the complexity of setting that up is largely handled for you by IRSA: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f...

Even the logging service should not be using the IMDS identity. The IMDS identity should be very much locked-down, something like AmazonSSMManagedInstanceCore if I recall correctly.

Because not everyone uses EKS?

Then, as alluded to in the last section, you can link IAM to an OIDC identity provider to hand your workloads temporary role credentials

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_pr...

> why not issue each of them identities of their own

Because the easy ways of doing this are not nearly as secure as the short-lived credentials provided by the metadata services in these clouds.

With 5-min credential expiry you can know that if a developer logs into the VM (for debugging or troubleshooting or whatever) after 5min they can no longer invisibily impersonate that role. This makes audit logging and similar much more powerful.

I've frequently seen long-lived credentials being used as an alternative and this way anyone who have ever logged into any VM with that key can invisibly impersonate that service forever.

You can do this by writing your own credentials manager but this is much more work, harder to make as reliable as the one provided by the IAAP provider and you still need a root of trust somewhere. Relying on the metadata service moves the root of trust to something ephemeral and managed which is highly valuable.

I'm the technical lead for IMDS. Without an IMDS, the alternative is usually hard-coding long-lived credentials, which is much worse.

We've looked a few times at different ways that IMDS could vend different credentials to different user-ids. We documented how to use local firewall rules, which you've linked to. This gives single uid restrictions, similar to filterd. We also have a BPF based tracer tool, https://github.com/aws/aws-imds-packet-analyzer which can monitor which user ids and processes are calling IMDS (and which version they are using).

Our next thought was to expose IMDS as a filesystem. That way ordinary POSIX filesystem permissions could be used to control which user ids could read which credentials, and it would even work on Windows. But our research found that security issues (in applications and libraries that customers run) that grant local file reading privileges are even more common than SSRF issues (in part this because many SSRF issues allow "file://" urls, so they become a subset).

We've looked at interfacing with the kernel keyring and the TPM 2.0 interface (and we now have Nitro TPM) ... but both are difficult to call to user space. The latter doesn't get user ID granularity, and the former is hard to coordinate with from the outside for revoking and rotating credentials.

Maybe consider having a simple config I can stick into cloud-init to restrict access to IMDS to a set of users? Seems like something generalizable enough to be very useful to the cloud ecosystem.

Thanks for your comment, it's interesting to hear what you guys have tried out.

Absolutely, long-lived credentials are worse. The only advantage to long-lived/file-sourced creds right now is granularity. Being able to lock them down to specific processes/UIDs and have a role per service. I guess what we want is the best of both worlds - short-lived per-service creds.

It appears to be a difficult problem to solve. I was wondering if something could be done with kernel modules.

Are you still working on this? Would really like to hear about future developments.

I don't undertsand why AWS insists on using network connections for security when there is a much better approach: virtual block device. Provide a tiny virtual block device, put filesystem on it, and generate credentials there, in json file format or even directly as sdk's ini file. If filesystem cache is a problem, put tar/cpio on block device directly and require equivalent of "tar xf /dev/vda".

This solves so much problems: security is trival (set file permissions), audit is trivial, no more timeouts, deciding whether to expose credentials to container or not is trivial (using directory sharing...), the client code is dramatically simplified.

why just now? there are many known attacks (SSRF[1] for example) that are based on IMDSv1. maybe i'm missing something but feels like this should have been done a long ago?

[1] https://www.cyberbit.com/uncategorized/aws-imds-v2-secures-s...

The migration isn't transparent, IMDSv2 has been available for a while and they have been pushing for it using warning notifications in the console. AWS deprecates things very slowly and usually avoid breaking changes. For instance, EC2 Classic stayed alive a very long time after they stopped selling it.

Over years, they moved from alerting to changing the default and will later remove IMDSv1 for new instances. Even then, they will not remove it for the previous generations and those may stay for years, the current oldest is M4 which I think is from 2015.

Because unlike some other providers, AWS understands the value of backward compatibility. They were the first to make the concept of simultaneously deployed versioned public HTTP APIs work seamlessly at scale, and the number of actually impactful API deprecations they have done is tiny. This is one of them.

Prior to this change, IMDSv2 could be enforced on an account, machine image, and launch API call level.

If you're interested in this topic, you might enjoy reading https://steve-yegge.medium.com/dear-google-cloud-your-deprec...

i assumed it's due to bc but does that mean that bc > security? question is how hard do they push/expose the risks of using soon-to-be deprecated tech

IMDSv2 was initially introduced after the capital one attack. but aws users will have to explicitly enable it and its not enabled by default.

now they are changing the default modes for instances launched from aws console using quick start process. and sometime in 2024 they will give an option to customers to control the default value for all run instances API calls from an account

IMDSv2 has been enabled by default since launch, and was made the default in the AWS SDKs at the same time. What's changing now is that IMDSv1 will be disabled by default. Until now customers had to disable it intentionally, either at the instance level, or by using IAM policies / SCPs to block calls still using IMDSv1 issued credentials.

I think we mean different things, it is enabled by default but I'm saying thats not the default IMDS version that metadata endpoint points to for `/latest/`

for example if you run below curl request on ec2 instances launched from aws console before nov 6th then you will get a response

```sh

curl http://169.254.169.254/latest/meta-data/profile

```

but if you run the same command on ec2 instances launched from aws console after nov 6th, then you will get an error that auth token is missing

IMDSv2 doesn't change the URLs, the "/latest/" in that URL is referring to the meta-data profile, rather than the IMDS version. IMDS is a lower level change in the authentication workflow to use a token (which is retrieved using a PUT request).

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance... has a good example.

If v2 requires a token, where is that stored given that the best place to store tenant-specific bootstrapping config is... IMDS?

You can generate a token on the machine and use a PUT request to set the token. The main difference with IMDSv1 on this is that you now need two requests (One PUT and one GET) instead of a single GET which protects against most SSRF.

The 2019 blog post[1] has fairly good explanations on how it works and the threat model.

[1] https://aws.amazon.com/blogs/security/defense-in-depth-open-...

If anyone is curious on how this can be abused, here's my "There's no place like 169.254.169.254 - (Ab)using cloud metadata URLs" slides [1] and talk [2] from 2019.

tl;dr - misconfigured reverse proxies allowed cloud metadata URL access across the bigger cloud providers.

[1] https://github.com/opsdisk/cloud_metadata_extractor/blob/mas...

[2] https://www.youtube.com/watch?v=vxReTpBCmh8

Reply on news.ycombinator.com