Show HN: Managing SSH Access to AWS EC2 Instances Using SSM | Hacker News Reader

Show HN: Managing SSH Access to AWS EC2 Instances Using SSM

github.com

131 pointselpy136 comments

Seem likes an awfully laboured equivalent of GCP's OSLogin [1]. To SSH into an instance:

  gcloud compute ssh <instance_name>

GCP takes care of generating a key pair. The GCP linux images configure OpenSSH and PAM to check you have relevant IAM permissions. IAP [2] is used in place of a bastion if the instance doesn't have an external IP address.

[1]: https://cloud.google.com/compute/docs/instances/managing-ins...

[2]: https://cloud.google.com/iap/docs/using-tcp-forwarding

EC2 Connect is OSLogin-esque: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...

Session manager doesn't require inbound access iirc. An agent on the machine handles the request via a connection to the SSM service. Your request for a session goes to the SSM service and a websocket connection is established from the machine you need to remote into. A cool solution.

I assume it also logs the access in the SSM audit logs.

I'm pretty sure this only works if you use images with Google's sauce added, which means your instances run Google daemons that do things like network, user, and ssh config.

Correct albeit no Google daemons, just OpenSSH server, PAM and NSS configurations:

https://cloud.google.com/compute/docs/oslogin/#how_os_login_...

That's good to know.

...But if you use an image with Google's sauce, there are Google daemons running on it, just not for this.

https://cloud.google.com/compute/docs/images/install-guest-e...

You can also use ec2-instance-connect[0] to enable users to generate ephemeral keys that are valid for a few minutes, authenticated with IAM. The benefit is that it's all native after you've pushed your keys using either aws-cli or mssh.

It has more or less the same benefits that SSM has, and you can use the same method with ProxyCommand to establish authentication before connecting. You can also chain it using ProxyJump to use a bastion host. Wrapped in a tool like aws-vault you can quite easily work with 2FA and such.

[0] https://aws.amazon.com/about-aws/whats-new/2019/06/introduci...

Doesn't that still require the instance to have a public IP? I think the original solution doesn't even need a public IP.

No, but it requires you to either have some kind of peering, Client VPN Endpoint or a bastion host. I prefer a VPN or bastion host, since you add another layer of authentication in between. With Client VPN Endpoint you can also limit network access on a per-user basis to further lock down things.

Does ec2-instance-connect use the generic system account or does each AIM user/role get it's own instance account? Asking from an auditing and logging perspective, as in is there a way to tie back a command which was run back to an individual human directly?

It has to use an existing system account, yes. You do lose a lot of auditing abilities this way, since you cannot capture the content. For me this is more about establishing a secure way to grant people access to instances, without something like LDAP or SSH CA.

LDAP sends passwords in cleartext over an encrypted channel, I don't like the idea of someone else on the host I'm connecting to being able to read my password. Naturally you can use key based authentication with it.

I know that some people are required by law or regulation to log all these things, but personally I don't see much benefit of such audit logs. All bets are off when someone has local access to a machine. And there's this whole debate about using MitM-proxies for logging and security and the potential security implications they pose.

All that said, step-ca[0] and vault[1] looks really promising to provide some kind of provider independent way to authenticate in a secure manner. You also have teleport[2] but I haven't looked much into it.

[0] https://smallstep.com/docs/cli/ssh/ [1] https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-cert... [2] https://gravitational.com/teleport/

Thanks for the links, Teleport seems especially interesting.

Yeah, we are required by contractual obligations to log these things or at least sudo commands. Don't disagree that it's less than useful but only so many things you can argue about. The logging we can do with auditd but we need unique instance users for that to be useful.

(disclaimer, I was a founder of ScaleFT - acquired by Okta)

This is exactly why many customers use Okta's Advanced Server Access: https://www.okta.com/products/advanced-server-access/

It does certificates as a credentialing mechanism, but also full separate accounts, lifecycle management of accounts and sudo files.

Shameless plug for a tool I build for ec2-instance-connect: https://github.com/nodefortytwo/amz-ssh

I think the advantage of not having to have a public bastion is pretty nice though. I think i'd consider SSM over ec2-instance-connect

We've been using SSH over SSM at work, and it's been great for tying access to IAM credentials.

Using the SSM tool as a ProxyCommand for OpenSSH is brilliant. I can SSH to an instance (turning on agent forwarding, etc.) or SCP files back and forth, all the same way that I'd do it with a regular host. But that host doesn't need to be exposed to inbound SSH traffic!

I was also able to get the ProxyCommand working in a Docker container, so I don't have to install the AWS CLI or the SSM plugin on my (Linux) host system, with an .ssh/config entry like this:

  Host i-*
    ProxyCommand docker run -i --rm -v $PWD:$PWD -w $PWD -u 1000:1000 -v ~/.aws:/tmp/.aws -e HOME=/tmp -e AWS_PROFILE <docker image ref> ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'

A few problems in practice:

* We often use our bastions to do port forwarding, e.g. to connect to an RDS instance inside VPC. As far as I know, we can't accomplish this with just SSH over SSM, so we still keep our bastions.

* User management: SSH over SSM doesn't do any user or key management for you (which makes sense). It establishes the SSH connection and you're on your own from there. We use Keymaker [1] to dynamically create user accounts on the EC2 instances, and populate SSH keys according to the user's IAM profile. This works fine, but maybe there's room for simplification.

* Connecting based on instance ids (`ssh i-...`) can be awkward. It would be amazing if we could alias these somehow, like set up a CNAME that points to `i-...`. I wasn't able to get OpenSSH to follow a CNAME and then use the ProxyCommand. Maybe it's possible, though.

[1] https://github.com/kislyuk/keymaker

I haven't used it or looked into what it supports, but I did remember this announcement which may interest you: "New – Port Forwarding Using AWS System Manager Session Manager": https://aws.amazon.com/blogs/aws/new-port-forwarding-using-a...

To make the ProxyCommand setup a little easier, perhaps you could use `aws ec2 describe-instances` with the appropriate filter? For example, the article I linked uses:

    INSTANCE_ID=$(aws ec2 describe-instances \
                   --filter "Name=tag:Name,Values=$INSTANCE_NAME" \
                   --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" \
                   --output text)

to grab the instance ID from the Name tag. You could set up a shell script as a ProxyCommand to do this automatically, although beware that ProxyCommand's %h will always lowercase the input...

It's hard to overstate how much of a difference this makes for managing people's access to servers without having to manage access keys. Especially if someone leaves the team/company you don't need to roll out new AMIs or run your puppet/ansible, it's just a `terraform apply` to update the IAM policy and you're done. Absolutely love it.

One thing that I found annoying through the web interface was the lousy support of special characters. Every now and then either some character combination or a big wall of text would make the browser-based terminal almost useless. I have yet to make time to configure the cli access which I don't think will have these issues.

Very funny, we received a ticket from a security audit a week ago asking us to get rid of our bastion and use SSM instead. Having never heard of SSM, being in a rush to a deadline and lacking skills with ops, we kinda panicked. 10min later, we had found the SSM interface in the AWS console, found it extremely easy to use and had deleted our bastion :)

Note: the README says "I found myself in a situation where I couldn't easily copy a file from my machine to the server". Note that you can use S3 for that (`aws s3 cp ...`), though you should be careful with permissions on your S3 bucket

You can always just pipe things over ssh. And now that ssm supports local port forwarding you can easily use netcat to send files back and forth.

Solving the SSH over SSM is sadly useless right now. The whole point of why we began using SSM was logging.

1) We soon discovered that using bash (and not sh that SSM signs you into) doesn’t format the logs correctly and makes them unreadable 2) Using SSH over SSM doesn’t log at all

Both of these problems are on AWS github but as always, they don’t respond.

Does SSM generate unique users for each login or does it use a generic user? The later would make any logging outside of SSM difficult since there's no way to tie it to a specific IAM user except time windows.

You can specify the behavior. There is no unique user for each login, by default you are dropped to generic ssm-user. It's possible to set up a log-in into specific user based on the IAM username.

> The later would make any logging outside of SSM difficult There is a way to tie IAM user to SSM. You have to request SSM permissions and AWS logs that. AWS logs pretty much every single API call so it's just matter of storing and auditing correctly.

>There is a way to tie IAM user to SSM. You have to request SSM permissions and AWS logs that. AWS logs pretty much every single API call so it's just matter of storing and auditing correctly.

If you're using a generic ssm-user (and going to ssh so there's no ssm logs of the commands) then wouldn't you not be able to distinguish the commands (say in auditd) of two people logging in at the same time?

I'm reminded of RDS which does not store enough logs anywhere to reconstruct which IAM user/role was running specific commands when using IAM Authentication unless you create individual users in the DB.

> If you're using a generic ssm-user (and going to ssh so there's no ssm logs of the commands) then wouldn't you not be able to distinguish the commands (say in auditd) of two people logging in at the same time?

But that requires auditing directly on EC2, right? (You can indeed sync the logs somewhere, but that diminished the beauty of out of box logging)

> I'm reminded of RDS which does not store enough logs

AFAIK, it's not possible with their current implementation to accomplish this. We just swallowed that it won't work and for the sake of security created seperate users.

This is an example we've been working that creates an Okta-managed user who can get in to a tagged EC2 instance using Session Manager. No bastion/sshd/security group ingress rules required. https://github.com/symopsio/terraform-okta-ssm-demo

I've been using this script for a while and it works great https://gist.githubusercontent.com/qoomon/fcf2c85194c55aee34...

Still need to figure out how to do this with an ephemeral fargate task. I'm guessing that I can build an image that adds SSM.

Otherwise my developers can't get a Django shell in production for emergencies.

Hasn't this been solved by using SSH certificates instead of keys?

I've been using SSH certificates for a while with great success. The only downside, in relation to SSM, is to have to manage your keys an CA, but it worth to avoid vendor lock-in.

SSM doesn't require you to manage keys or have a bastion/vpn/public access. It'll also log access attempts, in some cases log commands, and ties into IAM permissions (ie: these people can't access prod but can access test, etc.).

This is great. We usually overload the bastion server with NAT duties but SSM + hosted NAT makes bastion servers a thing of the past?

I've been rolling SSM out at work; 10/10 would do again.

that's pretty great. been using ssm for a while but wasn't aware of AWS-StartSSHSession, which solves all the quirks I had with ssm in the first place (like loosing the abilities of scp, tunneling and such).

Can anyone give any experience using this method vs. AWS Secrets Manager?

thanks!

Reply on news.ycombinator.com