Comment by gurchik | Hacker News Reader

Comment by gurchik | Hacker News Reader

Parent Full thread

I have been looking into SSM recently and I was a little confused by the setup instructions but after seeing your comment I read them again and I think I understand more now. I was trying to see how SSM could be used to eliminate the need for engineers to have SSH keys set up with instances.

  > Who should use Session Manager?  
  > ...  
  > Users who want to connect to an instance with just one click from the browser or AWS CLI without having to provide SSH keys.

Perfect, how do I get started?

  > Step 8: (Optional) Enabling and controlling permissions for SSH connections through Session Manager  
  > ...  
  > Create or verify that you have a Privacy Enhanced Mail certificate (a PEM file), or at minimum a public key, to use when establishing connections to managed instances. This must be a key that is already associated with the instance.

Uh oh, what's going on?

Well, I think I understand now that using SSH with SSM is optional. You can use the AWS Console or the AWS CLI to connect to a shell on the instance without using SSH. Then you don't need keys and even grants additional benefits like the ability to shell into instances without a public IP and without opening any ports. However, SSM+SSH has some advantages like SCP which I don't believe is supported by SSM alone.

Another thing I've been looking into is AWS CloudShell which is free. When opening CloudShell essentially a virtual instance is created within your VPC with Amazon Linux installed. You can then use this shell to SSH into instances. I haven't looked into it much because I would rather use SSM but I believe this could be used to essentially be an ephemeral bastion that is secure in that it doesn't accept public requests, it can only be accessed via the Console.

The trick is to mix SSM with EC2 Instance Connect using the `aws ec2-instance-connect send-ssh-public-key` command.

We use bastions to connect to RDS instances. The bastions aren't accessible from the internet; only via SSM. You can wrap up all of the steps in a shell script that calls `ssh`, or with a bit more effort, concoct a ProxyCommand script that does everything for you and makes e.g. `ssh aws-bastion` just work.

We have a script used as an SSH ProxyCommand that:

1) queries EC2 to find a bastion host based on tags (the bastions are in an ASG and can change)

2) generates an SSH key

3) adds the generated private key to ssh-agent temporarily (using the `-t` parameter to `ssh-add`)

4) sends the generated public key to the selected host using ec2-instance-connect

5) starts an SSH session using `ssm start-session`

Then a `~/.ssh/config` entry that intercepts connections for host `aws-bastion` and specifies the ProxyCommand (as well as keepalive and ControlMaster to make subsequent connections fast).

Adding the key to the agent temporarily is a trick since there's no other way to pass information from a proxy command to the outer `ssh` process, and I couldn't find any other hook. I've found at least one instance where that trick doesn't work: when connecting to a database from within IntelliJ's database tools. For that, I added an option to the proxy command script to pick a key already registered in the agent rather than generating a new one (e.g. `ssh-add -L | head -1`).

You are starting an SSM session yet using SSH keys? Can you explain that more?

SSM has StartSession which drops you into a shell, and StartSSHSession which is the equivalent of opening a tcp connection to an ssh server. When used as a ProxyCommand, the latter lets you do everything you'd normally so with an ssh connection, including tunneling. But you still need keys or some other auth mechanism.

Using CloudShell / the AWS Console for anything involved sounds painful though, among other reasons due to the 12-hour console session timeout. I suppose you might be able to use screen to mitigate it, but getting logged out right in the middle of doing something important seems unpleasant. The same thing makes AWS Sagemaker Studio unusable.

You can use the CLI and your terminal directly, as well as use SSH over SSM for proxying and SCP.

Reply on news.ycombinator.com