How to SSH Properly
gravitational.com
gravitational.com
- Setup and use ssh-agent. They make the life so easy.
- You can use Yubikey and pretty much every other solution out there with with OpenSSH now.
Sadly whatever library Spring Integration uses does not support them yet, as I found out the hard way. I just tried to search for it and couldn't find the reference but just dealt with it a month or two ago.
My #1 ssh usability tip: put this into ~/.ssh/config:
AddKeysToAgent yes
1. Don't put secret information on a machine where someone you don't trust has root.
2. Don't use agent forwarding unless you know you can trust the machine on the other end.
Since agent forwarding is not enabled by default, both of these seem pretty obvious to me.
Most "use cases" I've come across could have been solved "better" by simply using ProxyCommand instead. That was apparently too hard, though, so use of agent forwarding continued.
Since the introduction of ProxyJump a while back, however, it's now even easier to avoid using agent forwarding in most -- but not all -- cases.
So yes, be careful when using agent forwarding but, more importantly, as much as possible, just avoid using it entirely unless you absolutely must!
On both ends. Agent hijacking can happen on client or server, with an attacker present. And is there any machine you can 'trust' ? That's a big ask, and I think the modern 'zero trust' is fundamentally averse to the concept.
This gets said a lot, but doesn't forcing a prompt every time the forwarded key gets used mitigate this? SSH is not like surfing on the web with traffic flowing everywhere all the time. If you did not just now run any commands that are expected to invoke SSH, you probably don't want to answer yes to that prompt.
If you have more than one private key in your agent, then SSH will try each one sequentially. That can lead to mistakenly getting banned by monitoring software for what can appear indistinguishable to be an authentication attack. Not to mention the wrongness of presenting "any and all keys" to any host you connect to.
If you have multiple keys in your agent, you _really_ should manually configure what key to present using `-i` or `IdentityFile`. But you should also use `IdentitiesOnly` too.
If you specify -i or IdentityFile, the agent isn't used at all. You will have to type the keyfile password even if the key is in the agent, therefore making the agent useless.
This "feature" has been a big annoyance for me, since I like to use different keys per machine.
Maybe it depends on version or something? I've had the same config for years though over several very different OSes and presumably versions of everything.
I thought so as well, so the reply to this message surprised. I looked into it, and the following is the case on my Debian 18.04 machine running KDE:
With specifying the key through `IdentityFile` I can happily connect without a running ssh-agent. So it's true, that it can do without using the ssh-agent. But if the key has a password, it will prompt for it everytime it's used. I wouldn't phrase it "the agent isn't used at all", though, because for when ssh-agent is running, and it contains the key+password, it seems to happily use that agent, as it doesn't prompt for the password anymore.
Side note: If my understanding is correct, `IdentifyFile` lessens the need for consulting the ssh-agent as stated above, but, except from consulting it for the key password, the agent might be consulted for one more reason as well: Iterating over keys if using the specified one proved unfruitful. For this to stop, you'd have to specify `IdentitiesOnly yes` as well. But this I didn't test, so it's based on theoretical understanding only.
Edit: Oh, the last part was already explained in some other thread which branched of from this. So this post didn't actually provide some new insights, it seems.
Having a forwarded agent was how matrix recently got hacked, confirmation is a decent work around if you need to forward your agent.
I discovered this week that AWS doesn't support Ed25519 keys for EC2 and IAM users.
But at virtually no additional cost. I don't see your argument?
Bigger RSA keys are expensive to compute:
$: openssl speed rsa
sign verify sign/s verify/s
rsa 512 bits 0.000075s 0.000006s 13249.7 179054.0
rsa 1024 bits 0.000136s 0.000011s 7363.7 90816.2
rsa 2048 bits 0.000920s 0.000028s 1086.8 35897.8
rsa 3072 bits 0.002575s 0.000053s 388.4 18845.6
rsa 4096 bits 0.005716s 0.000093s 174.9 10777.2
rsa 7680 bits 0.054218s 0.000304s 18.4 3287.0
rsa 15360 bits 0.283036s 0.001182s 3.5 846.2But that argument shouldn't be applied for all use cases, such as personal key management, or keys that are used for days/weeks/months at a time.
I tried switching from a 2048-bit to a 4096-bit key to control a modestly sized VPS (~4GB RAM, 2CPU) and scp file transfer speeds plummeted.
I have a symmetric 1GB FTTH connection and I'm used to everything being pretty quick. Using a longer key was like a return to dial-up speed. If you don't plan to transfer large files or directories you can safely ignore it, but I bet your patience will run out pretty quickly if you do.
(Unless you are saying you somehow convinced SSH to use a symmetric cipher with a 2-4k key size...?)
> ...
> You can use Yubikey and pretty much every other solution out there with with OpenSSH now.
Unfortunately, you can't use Ed25519 keys unless you have the latest/newest model(s) of Yubikey and that Yubikey came with at least a specific firmware version (5.2.3, IIRC). Even if that weren't so, there's still loads of popular / widely-used software that isn't even close to supporting Ed25519 (yet) but has supported RSA for years and years (and will continue to for the foreseeable future).
I'd like to be able to "upgrade" to an Ed25519 key but I can't come up with any real reason to expand my collection of Yubikeys any further (I think I'm up to nine, at last count, and the three newest ones haven't even been opened!). I don't plan on purchasing another Yubikey until four or five of the ones I already have decide to call it quits, so I won't be using an Ed25519 key anytime in the next couple of years, at least.
Even if I could start using an Ed25519 key with my Yubikeys today, I'd likely still have to keep an RSA key around for the "legacy" stuff, in which case I might as well just save myself the trouble and stick with RSA for a while longer.
A private RSA (2048+) key on a Yubikey is still much "better", IMO, than a private Ed25519 key sitting on one's hard drive.
In an ideal world we could all just switch everything over to Ed25519 at about the same time and then retire RSA shortly afterwards. It's gonna be a long time before everyone has ditched RSA keys and moved to Ed25519, though -- RSA isn't going anywhere anytime soon!
It seems to work great with all the yubikeys I’ve tested with.
I am a bit confused. Aren't yubikeys able to store multiple keys on them? Why do you have to use so many? Why not instead e.g. a LUKS encrypted usb flashdrive?
Well, I started out with a Symantec VIP Yubikey many, many years ago. Then, MtGox sent me one for 2FA after some compromise there. Later, I bought a "full-featured" Yubikey for use with GPG and SSH on my desktop / workstation. After that, I got two Nanos so that I could leave one, permanently, in each of a pair of laptops. Shortly after that, I decided I should have a spare -- you know, just in case.
At some point, I received another pair of Nanos as free replacements due to one of the issues that Yubico had (weak RNG or something, I think?). Those two and the "spare" are the ones that are still unopened in the package.
Finally, (I'm not really sure why but) I bought a pair of the "GitHub" U2F-only Yubikeys when there was some special deal going on. Now that I think about it, I don't think I've ever even used them either.
The only ones that I really use are the ones that I leave in my workstation and my primary laptop (primarily used for SSH but, to a lesser extent, for signing git commits and unlocking KeePassXC databases as well).
> Why not instead e.g. a LUKS encrypted usb flashdrive?
Oh, I've got probably a dozen or so of those also!
I should look into setting it up with more things at home.
I've been looking at MFT solutions recently (think fancy SFTP servers for file exchange) and the level of support is just plain bad. Out of roughly 20 products, I think there are maybe 2 that had ed25519 support.
Similarly with Java products that need to use SFTP to transfer files (think SuccessFactors, Dell Boomi etc.) These typically use JSCH which has not been updated for years.
facepalm
https://developercommunity.visualstudio.com/idea/462263/cant...
https://feedback.azure.com/forums/223579-azure-portal/sugges...
This is only true if you consider pure brute force as the only way to "break" RSA 2048. While as of right now there is no hard evidence there has been plenty of hearsay that some of the 5 Eyes have had tools for years that can drastically reduce the brute force complexity needed for RSA 2048 keys.
There is also really no such thing as being "paranoid with no gain" when it comes to computational security, since digital assets can be stored indefinitely and compromised in the future with more advanced computational power or techniques. On the contrary, given the computational power of your average laptop/server these days there's really no reason to _not_ use 4096 keys, unless you are operating on FAANG scale.
So maybe the title shouldn't be "I like ssh certs" but "ssh security for organisations", but your point still stands.
It's the name of the TOTP PAM module they use, which was created by Google: https://github.com/google/google-authenticator-libpam
I'm considering doing a future post on how to set up U2F for SSH with hardware devices (like a Yubikey) as well. I'm curious if you have anything else you'd like to see on this topic.
That's not a coincidence, of course. It's purely sales and marketing. To be clear, the ultimate goal of this blog post is simply to get you or your employer to give them some of your money -- that's it.
(To be clear, giving an external entity full control over authentication and authorization of your critical servers is definitely not doing "SSH properly", IMO!)
ssh-keygen -t ed25519 -b 521 -f .ssh/id_foo_ed25519
cat id_foo_ed25519.pub | ssh foo tee -a .ssh/authorized_keys
ssh-add -K ~/.ssh/id_foo_ed25519
Host foo
HostName foo.bar
User me
StrictHostKeyChecking yes
IdentitiesOnly yes
IdentityFile ~/.ssh/id_foo_ed25519Put the target hosts in a file e.g. ssh_ducks
while read -r i; do ssh-copy-id -i /path/to/ops_ed25519.pub <user>@$i -f; done <ssh_ducks
Otherwise, for small to medium scale SSH access, I prefer to use jumpserver (open source 4A bastion / remote access gateway) behind VPN (WireGuard of course, for now, self-hosted or Tailscale), a lot easier to deploy in comparison to Teleport, user-friendly UI for average users.
[1]: https://github.com/openssh/openssh-portable/blob/master/cont...
Host foo
...
Port 2019
ssh-copy-id -i id_foo_ed25519.pub foo@barThe most important thing is to look at your attack surface realistically and plan accordingly. Just blinding putting in an SSH cert authority of an unsecured-non-redundant ldap cluster is worse than no key management at all.
2) Firewall off all but a few bastion hosts and ProxyCommand through those. Immediate revocation can be assured by updating the CRLs on those specific hosts.
https://github.com/Netflix/bless
there's also cashier
https://learn.hashicorp.com/vault/secrets-management/sm-ssh-...
It's Hashicorps most mature tool and should be in everyone's toolchain.
Vault truly shines in its ability to provide ephemeral credentials. It can even "package" arbitrary secrets for you, and ensure that they are only ever retrieved once.
It's a fantastic piece of software, one that more companies should be using, but it is not a push button deployment. It is borderline impossible to implement in many 'traditional' organizations.
Those credentials should not be generated, stored or provided centrally in the first place.
-z 20200402
I've also never had an issue with key management, like the article suggests is common. Maybe I'm not the target audience though.
But generally, certificates have a number of advantages: they expire quickly (you can have one-shot certs), they contain your full name, email address, or whatever you wish via metadata, this allows to implement RBAC on top of SSH. They address "trust on first use" issue, and they're easy to synchronize with other protocols, i.e. you can have the same CA issue certificates for both Kubernetes and SSH (with identical RBAC rules).
When implemented properly, they're actually easier to use. The whole world uses certificates to do online banking and shopping, there's no reason not to have the same seamless experience for SSH.
You can do this by adding a line like below into ~/.ssh/config
Host alias domain.com HostName domain.com User aUser
This allows you to just write `ssh alias` and it will convert it to ssh aUser@domain.com.
IdentifyFile /path/to/key/file
when I have different hosts that require different keys to access, this is very useful so I don't have to specify them every time
and agent forwarding is useful as well, for example, I can push to github from a vm using my laptop ssh key and I don't have to generate an additional key for the vm
ForwardAgent true
with the new Windows terminal, I setup different tabs to be SSH to different machines based on my .ssh/config file
{ "guid": "{1c9b268e-7606-4a07-b097-d8bc62fb5207}", "hidden": false, "name": "ubuntu.localdomain", "commandline": "ssh.exe me@ubuntu", "colorScheme": "Elementary" }, { "guid": "{b7161acc-8235-4283-9a85-e93df3d09125}", "hidden": false, "name": "db.localdomain", "commandline": "ssh.exe me@db", "colorScheme": "Ubuntu" },
Windows Terminal with a very nicely define ssh config is a killer combo.
If you have multiple keys in your agent, you _really_ should manually configure what key to present using `-i` or `IdentityFile`. But you should also use `IdentitiesOnly` too.
So definitely have a custom ~/.ssh/config
Possibly a different key for each machine, especially public because your public key can be used to identify you.
My only wish is that there was a way to define a backup hostname/IP of the first is down (my laptop has different IP on local WiFi, local wired, remote) and a way to specify multiple names for a host (on PC, I usually use NAS or the full IP address; on my phone I usually just use the last octet or lowercase nas); it would be nice to have the same file configured and in use across all my devices, with only a single configuration block for each.
I thought this was going to be about Yubikeys or other HSMs.
Exactly how big your team needs to be is a judgement call, but I think a reasonable threshold where you really should start thinking about CAs for SSH access is around where it's no longer possible for the dev ops people to be aware of what everyone who needs production access is working on.
I still don't think people should be using SSH CAs for anything that isn't a short-lived, single-use bastion/proxy key. They're on the right track with Teleport; not so much with this standalone article.
The people who worked on Linode Library in the beginning were engineers, not social media people nor content marketers. The person who wrote its first CMS is a former nuclear submariner and Perl guy with like nine followers on Twitter. I’m sure you imagine a huge meat grinder spam operation paying by the word, but basically one engineer thanklessly wrote more than half of it in the beginning and went on to a distinguished career at 10gen/Mongo, not sitting on Mechanical Turk. I’d bet DO’s operation is identical and exists for the same reasons, and they’re just as proud of it as I am my extremely limited contributions to an offering that objectively improves the operations discipline.
It’s annoying to watch valuable work that expands the profession get dunked on and called spam by a snide comment for absolutely no reason.
There’s no reason to be so cynical about things in a hot take, particularly when it was an irrelevant shot in context. Not everyone is blessed with innate knowledge nor a gift for studying man pages, and those docs are useful checklists to make sure I didn’t forget anything with common software like ntpd to this day in my own, principal level, career. People learn Unix operations by buying a Linode. For a $10/month VPS you also get a bunch of tested, free guides that work on them and teach you how to do things. They’re not paywalled to customers and show up in your Google results. I know. The abject horror is almost unbearable.
I'm probably misunderstanding what you meant, but between those two I'd say MITM is a much higher risk than someone being able to break everything to the point where you need a yubikey/HSM to protect your keys.
export SSH_ENV=$HOME/.ssh/environment_${HOSTNAME}
function start_agent {
echo -n "Initialising new SSH agent... "
touch ${SSH_ENV}
chmod 600 ${SSH_ENV}
/usr/bin/ssh-agent -s | sed 's/^echo/#echo/' > ${SSH_ENV}
echo "succeeded."
. ${SSH_ENV} > /dev/null
}
# Don't use Gnome keyring for SSH
if [[ $SSH_AUTH_SOCK =~ /run/user ]]; then
unset SSH_AUTH_SOCK
unset SSH_AGENT_PID
fi
# Source SSH settings, if applicable
ssh-add -l > /dev/null 2>&1
if [ $? -eq 0 -o $? -eq 1 ]; then
echo "SSH Agent found."
else
echo -n "Looking for SSH Agent... "
if [ -f "${SSH_ENV}" ]; then
. ${SSH_ENV} > /dev/null
ssh-add -l > /dev/null 2>&1
if [ $? -eq 0 -o $? -eq 1 ]; then
echo "found pid: ${SSH_AGENT_PID}."
else
echo "not found."
start_agent;
fi
else
start_agent;
fi
ssh-add -l
fi
Edit: I see it's been mentioned elsewhere here.
I didn't want or need to wrap ssh-agent. I'm fine with using it and ssh-add by themselves. I just wanted to automatically find and re-use an existing agent rather than spawning a new one.
Sometimes I'm the one remote, logging back into my workstation. I've previously had Gnome keyring helpfully pop up a window so that I can unlock a SSH key... meanwhile I'm just sitting there (SSH'ed into my workstation) wondering why my outbound SSH command is just hanging.
Free and open source, made by the authors of the article.
And, just so everyone is clear, your employer.
It's a nice practical guideline with all the basics covered.
https://github.com/bitnomial/aws-ec2-knownhosts
Right now, this is pretty specific to our use of it, specifically our use with Terraform EC2 instances. We'd happily suggest changes to make it more generic. But you can see the parsing logic there.
(Disclaimer: I work for AWS, but opinions are my own and not necessarily those of my employer.)
1. Network forwarding (local-to-remote, remote-to-local, dynamic, and unix socket support). The link you supplied mentions only local-to-remote.
2. Agent-forwarding (don't worry, I have confirmation-on-every-use, see my other comment: https://news.ycombinator.com/item?id=22753590). This I use all the time to be able to authenticate to other SSH servers and git hosts. This is a must-have for me now for remote development and pair-programming.
3. sshfs, sftp, and rsync. Again, absolute beauties at the job of managing files remotely.
This is just off the top of my head, because I use these features day-to-day, but there's many other nice things ssh does that have nothing to do with the underlying connection or authentication method.
With Agent forwarding any authorised machine you connect to (and thus anyone who controls that machine) gets to use your credentials for the duration, to my knowledge no general purpose clients give you feedback on this usage, so you won't know if this happened - if Agent forwarding is enabled it can be used even if you never use it. With ProxyJump the intermediate doesn't get any view of your credentials, not even whether those are the same credentials you used for that intermediate host, or different. It is only enabling you to connect to a host it can reach that you can't reach directly and nothing more.
I'm afraid you're mistaken here.
ssh-add(1) and ssh-askpass(1) support confirmation per-use: https://man.openbsd.org/ssh-add.1#c
The agent I personally use — gpg-agent — also respects this protocol and asks me to confirm each use: https://www.gnupg.org/documentation/manuals/gnupg/Agent-Conf...
Even if it did not, I can always configure gpg-agent to never cache the passphrase, thus making it ask me for the passphrase every time. This would also adequately serve as a means of confirmation, but thankfully this tedious option is not necessary, because of the above.
> Prefer ProxyJump where applicable.
I already do, but it's pretty orthogonal to agent-forwarding. The uses of agent-forwarding I mention in my original comment simply cannot be served by ProxyJump.
I think Fido2 mode does support it but I don't know whether I trust Fido2 fully for SSH.. Especially because it seems to still store a private key locally, which defeats the whole purpose of using a hardware key (I saw this here: https://developers.yubico.com/SSH/ ). The whole idea is to not have the private key anywhere but inside the token itself. It's also a bit early in development where RSA has proven itself. It has some known weaknesses (including high quantum computing vulnerability) but they're well understood.
FIDO was conceived for the Web, and so the SSH authentication model wasn't a core consideration in its design.
In SSH authentication always goes like this:
Client: "I can prove I'm Alice because I can do X" Server: "OK, do X" Client: (Proves they are Alice by doing X)
OR
Client: "I can prove I'm Alice because I can do X" Server: "Not good enough. What else?" (Repeat with a different method or give up)
Whereas WebAuthn (and U2F and other models FIDO was proposed for) have authentication go like this:
Server: Prove you are Alice, if you are you can do Y or Z Client: (Does Z)
In FIDO there's a magic "cookie" used, in theory this could be just a "serial number" for a credential, but in reality for a typical FIDO dongle (such as Yubico's Yubikey) it's actually your private key for that site, encrypted using a symmetric key known only to that Yubikey. This way your Yubikey can unlock a genuinely unlimited number of sites, each with completely fresh credentials that can't be correlated, without infinite flash storage. A site says "Here's that cookie you gave me, now prove you're GekkePrutser" and your Yubikey consumes the cookie, decrypts it to get a private key, then uses that private key to prove you're GekkePrutser.
But as we saw in SSH the authentication doesn't happen in that order. The Security Key needs the cookie first, but the remote server is waiting to hear how you'll prove who you are first, it's a stalemate.
To break the stalemate the cookie is stored on your local system, so that can be fed to the Security Key, and then things can proceed.
Down the road apparently the OpenSSH team will add support for FIDO2's "usernameless" behaviour which doesn't need a cookie but does consume limited resources on the Yubikey. If you just can't accept the idea of storing the opaque cookie value because you know it has your private key encrypted inside it, then that mode will suit. It's also useful for "road warriors" who are willing to trust other people's computers briefly but don't carry their own. But that isn't finished today and it seems they don't expect it to be the usual way this is used, since it requires a FIDO2 (not just FIDO) Security Key.
It does, but you need a more recent Yubikey with firmware 5.2.3 or later: https://support.yubico.com/support/solutions/articles/150000...
Does that mean that GnuPG is overly paranoid? Or ssh-keygen's keys potentially insecure? (I really hope not). There must be some good explanation for this huge speed difference.
It really just waits longer.
Certificates can do a lot more than authorized keys can, like enforcing the use of specific principals, commands and options and embedding that information into the file itself without needing to modify each server's SSH configuration. They're also self-contained and will still work in situations where some external service providing a list of keys goes down. I've been on the rough side of a huge LDAP outage which prevented necessary access to the infrastructure to fix it, and it was a horrible experience. There's none of that problem with certificates as long as you make sure you have one which is currently valid.
I'm also generally of the opinion that it's safer to enforce the use of authentication which expires by default rather than relying on some external process to do that for you.
Look at it in terms of building in a decision at compile-time rather than at runtime. With AuthorizedKeysCommand, you're running something just-in-time on an SSH login to determine whether something should be allowed to proceed. With a CA and a process for issuing certificates, that decision is made at the time the cert is issued and then the cert is good for the duration it's issued for. It's entirely self-contained as sshd itself is making the decision about whether the cert is within its validity period or not.
It's obviously a decision that people can make based on their own infrastructure, but my opinion is that the compile-time model is more reliable as it's a fully self-contained system and doesn't rely on an entire fleet of servers being able to connect back to an external service at runtime to determine whether you should be allowed to log in. That sort of thing invariably comes back to bite you when you really _need_ to be able to log in and you can't because the external service is down.
If you have a certificate which expires within a day by default then an unsuccessful revocation is no longer a huge cause of stress. In the worst case, you lock down access to your bastions and disallow the issue of any future certificates for that user. Within a day, any potential threat from that certificate has vanished. This seems preferable to having a mandatory requirement of an up-to-date revocation database which is synced everywhere.
I'm also not sure it's easier to "lock down access to your bastions" and wait out the certificate expiration instead of having a certificate revocation database. Although OpenSSH does not provide a mechanism to distribute the revocation list it seems trivial to add a certificate to the revocation list and distribute it in an automated fashion.
Lastly, since you have to both lock down hosts and wait out the expiration, does that not constitute a fail-open system? I really don't think an expiration date mechanism makes this a fail closed system. Either method requires manual intervention upon compromise.
The overarching reason isn't really a question of "helping users" as such, although I would strongly encourage making the certificate issuing process as quick and easy as possible to encourage adoption and reduce pushback. The people it really helps are security teams and organisations as a whole who can now have more confidence that they haven't left holes in their infrastructure which can be exploited by bad actors. It also checks a lot of boxes for auditing, compliance and reporting purposes which are huge positives in a corporate environment. If you're able to say "yes, disgruntled former employee X had a certificate that would have given them access to all these servers, but it expired three days ago" then that's a lot better than saying "X has a certificate that gives them access to all our servers, but we _think_ we've blocked it from being used everywhere".
Overall, I agree that the model does lend itself better to things like access to critical production infrastructure (where access should be the exception rather than the rule), but in my opinion it's a good practice to get into for access to everything. The ability to log that a certain user requested a certificate at a certain time and then link that to exactly where the certificate was used (via centralised logging, for example) is incredibly powerful.
You're perhaps correct that both do constitute fail-open systems at first. The difference is in the vulnerability period - with an expiring certificate, that ends at a fixed point in the future. With a certificate that has no expiry, that period never ends until such time as you rotate your CA and force everyone to get a new certificate - something which is also far less of a burden when your certificates expire every day by default and you have a process for getting a new one, incidentally.
You could make keys valid for only a minute and it wouldn't add any security, as only seconds are needed for a malicious action to take place.
https://stuff.purdon.ca/?page_id=70
https://sigg-iten.ch/learningbits/2014/11/13/first-steps-wit...
I've still got one here, but it's probably been 10 years since I've used it. I haven't forgotten how much of a PITA it was to get it working, though!
resource of how to: http://wolf-tm.com