Down the rabbit hole: Implementing SSH port forwarding over AWS Session Manager | Hacker News Reader

Down the rabbit hole: Implementing SSH port forwarding over AWS Session Manager

joinformal.com

59 pointssaligrama13 comments

If you just want to enable ssh to ec2 instances (through SSM) using ssh i-… you can add the following lines to your ssh config

https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0...

host i-* IdentityFile ~/.ssh/id_rsa TCPKeepAlive yes ServerAliveInterval 120 User ec2-user ProxyCommand sh -c "aws ec2 start-instances --instance-ids %h ; aws ec2 wait instance-running --instance-ids %h ; aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user %r --ssh-public-key 'file://~/.ssh/id_rsa.pub' --availability-zone $(aws ec2 describe-instances --instance-ids %h --query 'Reservations[0].Instances[0].Placement.AvailabilityZone') ; aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

This will also allow VSCode remote development.

My variation is to use a custom script as `ProxyCommand` that resolves private route53 DNS names to instance ids, because remembering instance IDs is insane.

Mine is to run a Tailscale node on a tiny ec2 instance. Not only enabling ssh but direct access to database instances, s3 buckets that are blocked from public access etc

How are S3 buckets blocked from public access? I mean I know there is literally a “Block public access” feature that keeps S3 buckets from being read or written by unauthenticated users. But as far as I know without some really weird bucket ACLs you can still access S3 buckets if you have the IAM credentials.

Before anyone well actually’s me. Yes I know you can also route S3 via AWS internal network with VPC Endpoints between AWS services.

In general, condition keys

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...

And https://docs.aws.amazon.com/service-authorization/latest/ref...

Specifically the vpce one as the other poster mentioned but there's other like IP limits

Another way is an IdP that supports network or device rules. For instance, Cloudflare Access and Okta you can add policies where they'll only let you auth if you meet device or network requirements which achieved the same thing

> Specifically the vpce one as the other poster mentioned but there's other like IP limits

IPs don't cut it to prevent public access. I can create my own personal AWS account, with the private IP I want, and use the credentials from there. There's really just VPC endpoints AFAIK.

You essentially add a policy that limits the access to only come from your VPC endpoint.

I run an EC2 instance with SSM enabled. I then use the AWS CLI to port forward into the 'private' database instance or whatever from my desktop. The nice thing about this is it's all native AWS stuff, no need for 3rd party packages, etc.

Very excited about the fork's improvements given the lack of responsiveness in core repo. I've hit some of these bugs including the registry issue. Thank you for open sourcing it.

I've used a combination of ProxyCommand directive in ssh config + a script it calls w/ the `%h` (host) to unpack what the correct instance-id is (like @galanwe). For the proxycommand, you can embed an `aws ec2-instance-connect send-ssh-public-key` for pushing a key valid for 60s followed by activating the SSM session.

The downside is it adds ~20-30s delay in connection due to the API requests, but if you're making repeated rapid requests to same instance, I recommend looking into ssh's ControlPath, ControlMaster and ControlPersist to keep a longer lived session that's re-used for client re-connections (ref: https://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing...)

[Edit to add that I've hit the registry bug myself]

Funny to read this after seeing Fly.io’s recent blog about VS Code’s remote SSH agent: https://fly.io/blog/vscode-ssh-wtf/

I’m surprised this hasn’t come up more frequently. I’d expect a lot of security products to flag it as very much looking like malware.

The System Manager Agent on EC2 instances is meant to have enough privileges to manage fleet of instances at scale. System Manager Session Manager (yes that’s the real name) is a part of it.

What do you suggest as an alternative? Session Manager is IAM controlled and much more secure and easily managed than the traditional means involving direct access via SSH and opening a port through the security group for a certain IP address or a jump box server.

Was wondering if it's possible to use sshuttle with SSM but didn't realize SSM had its own mixing protocol.

An SSM client library is super helpful for embedding the tunneling into the client app. At a previous employer, we'd access jumphosts from AWS Session Manager and GCP Identity Aware Proxy and were looking to embed that connection flow into some cli management utilities to ease use. At the time, we ended up starting the tunneling processes separately inside the code but it would have been much nicer and easier with a library.

Does using SSH this way solves the Ctrl-Y problem in AWS Session Manager, which kills the session?

Reply on news.ycombinator.com