627 karma · joined April 4, 2007
Heads up, Joe is using our hosted product which you can find at https://smallstep.com/sso-ssh/. It's $3/host/mo (actually $0.0041/host/hr) with a 30 day free trial.
The linked blog post explains how to do single sign-on for SSH yourself using our open source toolchain. Choose your adventure!
ACME support in `step-ca` means you can leverage existing ACME clients and libraries to get certificates from your own private certificate authority (CA). This is particularly useful for:
* Using ACME in production to issue certificates to workloads, proxies, queues, databases, etc. so you can use mutual TLS for authentication & encryption.
* Simulating Let’s Encrypt’s CA in dev & pre-production in scenarios where connecting to Let’s Encrypt’s staging server is problematic.
We're supporting the `http-01` and `dns-01` challenge types, so you should be able to get certs using any ACMEv2 client that supports one of these (most do).
* More info and examples on our blog: https://smallstep.com/blog/private-acme-server/
* step-ca is on GitHub: https://github.com/smallstep/certificates
We've been working on this for a while so we're psyched to release it and see how people use it! Let us know what you think!
We also have a one-time-token mechanism that you can have some trusted infra like Puppet issue to hosts as they come up. The token includes the specific name that you want bound in the cert. It can only be exchanged for a cert with that subject.
Super secretly: we also have a whole policy language and enforcement engine that we'll eventually get around to doing something with and would address this issue pretty comprehensively.
Edit:
After reading your comment again I think I still might be misunderstanding your use case. Are you talking about having principals like "frontend" and "backend" and then having RBAC that says "mike can get a cert for frontend"?
There’s a difference between being respectful and walking on egg shells. I can respect you and still say, bluntly and to your face, that you’re wrong. In fact, I think not doing that would be disrespectful.
The title was tongue-in-cheek. It communicates a concept compactly. It’s direct and draws attention. It’s hyperbolic, but hyperbole is a legitimate literary tool. I am sorry if people are taking it too literally and somehow making it about them and their insecurities and being offended. But I think that’s as much on them as it is on the title.
This word police thing is sort of annoying and pedantic. If you actually read the article it’s unboxed and explained carefully and fully. That’s what matters.
> chef/puppet/etc ... put just your pubkeys somewhere (S3, eg) ...
Yea, cool, and now you have a second authentication system that you have to manually administer to onboard and offboard people. And you have permanent long-lived single-factor credentials on a bunch of (possibly unmanaged) endpoints, that are easy to exfiltrate, and are a common target for attackers if a laptop is lost/stolen/compromised.
You haven't addressed host name reuse or host rekeying which, again, are real operational problems.
And somehow running and administering chef/puppet/etc or this S3 bucket is easier than running a simple certificate signing service connected to SSO that requires basically zero maintenance once it's setup? Argumentative, at best.
> Next problem is that openssh-style certs do not support cert chains ... recover from compromise.
Since only hopeless organizations don't have config management, we can solve this there. SSH PKI is simple and doesn't have intermediates. But you can configure multiple roots. To rotate or revoke a compromised private key you'd simply deploy a new root using cert management and issue new certificates. In a non-emergency situation you can slowly roll certificates by having two trusted roots for a transitional period while everyone gets new certs.
> You have to secure the CA!
You have to secure your configuration management that's deploying public keys everywhere and your authorized keys files on every host! Pubkey distribution has a bigger attack surface area and the same exact compromise consequences as a CA. This is not even a contest.
> You have to properly authorize signing requests!
You have to authorize requests to deploy public keys!
There are tools for this. What's the hard part here? Verifying an OIDC identity token? Mapping a token subject to an SSH principal?
Worst case, you could keep whatever crappy manual process you have for authorizing public key distribution and issue a certificate instead.
> These things fall into the realm of hard.
Ok, I guess? This is all relative. It's easier than pubkey authentication at any non-trivial scale. There's a difference between something being hard vs. being unfamiliar.
This is one of a handful of problems we're working to solve / streamline for an SSH product. We might even open source this bit, but not sure yet. If we don't, it still probably wouldn't be $hundreds/$thousands per month unless you have a huge org. If you're interested at all I'd love to talk more about this -- what your requirements are, whether you want hosted/not hosted, whether you want to pay at all / how much you'd be willing to pay, etc. Easiest thing to do to stay in the loop is send us your email at https://smallstep.com/sso-ssh/ by "requesting early access" and we'll reach out to schedule a convo & keep you updated as we make progress. Or just watch our blog! :)
I still think it's harder to do this than it is to use certs, since you'll need to build some tooling to securely distribute keys. But that would vary by environment, for sure, and it's conceptually simpler for folks who don't know how certificates work I guess.
Do you know when AuthorizedKeysCommand was added to sshd? Is it available in most distros?
It's too bad there isn't a RevokedKeysCommand! That would make certificate revocation a lot more flexible. I wonder how hard it would be to add.
> medium/big companies
My goal (or our goal at smallstep) is to build tools to make this easy enough that it makes sense for small teams. If you have one client and one server, pubkey authn will probably always be easier. And as long as it's the default, certificate authentication will always require some amount of configuration. But I think good tooling with baked in defaults to encourage best practices can make this easy and quick enough to deploy to be a good idea for most non-trivial SSH deployments.
However, much of this is aspirational at the moment. So your point remains. The point of the post was to raise awareness and start a discussion about what's needed to make this feasible for pragmatic people.
> dedicated sysadmin team
I don't think a dedicated sysadmin team should be necessary. On the contrary, with the right tooling, and unless you're just completely punting on any rational management of SSH keys, I think certificates are easier to administer.
> potential attackers are not sophisticated enough to target SSH access
The more common attack vector is a lost/stolen/compromised laptop or a private key that's accidentally committed to a repository. This is a very real vector. It happens in real life on a regular basis. Since certificates expire a lost/stolen machine or committed private key will become worthless for accessing internal infra very quickly. So it reduces the attack window, which is a real and significant improvement. For a compromised machine, SSH certificates aren't a cure-all, but they're at least as good as pubkey. They make it easier to keep private keys off disk, providing better cover. And they make it harder for attackers to exfil private keys, cover their tracks on the compromised laptop, and do bad things with them later.
> You might even lock yourself out of your servers
You could do this with pubkeys too. If this happens, cloud VMs typically have a virtual serial interface you can use to remediate. You can leave pubkey authn enabled and still use certificates day-to-day, so you could keep a backup pubkey available for a few select admin users that you trust. Alternatively, you could issue a long-lived certificate in a secure place for use in an emergency. If this is the thing that's preventing you from trying it out I hope these ideas get you over the hump! If they don't, I'd love to hear why!
> they are just too abstract to bother for small companies
This is a very common problem with security stuff :(. That's why I think the headline feature here is "SSO for SSH". It's an operational improvement and better UX. The security benefits come for free.
Yea, fair... there's probably enough to cover for a whole follow-up on nuts & bolts and these sorts of considerations. This post was more about raising awareness and it was already super hard to keep it to 3000 words :P.
I think I can at least partially address all of these points though.
> Do all OS distributions have new enough openssh, is it ever built with missing support by default?
I haven't done an analysis but certificate authentication was added in OpenSSH 5.4 in March, 2010. So it's almost ten years old. So yea, I think all reasonably current distros have support at this point.
> what extensions are necessary in the certs to be trusted for ssh or the CA cert (is your average test CA cert going to be valid for ssh without tweaking)?
To clarify, SSH doesn't use X.509 like TLS/HTTPS. OpenSSH invented its own certificate format.
The docs for `ssh-keygen` have pretty good info on this.
* https://man.openbsd.org/ssh-keygen.1#CERTIFICATES has basic cert info * https://man.openbsd.org/ssh-keygen.1#O has info on some cert options / extensions
If you have experience administering nix boxes & SSH the options / extensions should look pretty familiar. Basically, it's a lot of the same stuff you'd ordinarily put in config files on each host, but instead you're baking it into certificates.
> Do any (or all?) cloud server products regularly accept cert based configuration just as they accept public key lists?
As far as I know, none of the cloud providers work with SSH certificates out of the box. It's kind of weird, actually, because it seems like super low-hanging fruit and would be a better user experience. Instead, they use public key authentication and deliver pubkeys to an authorized keys file for you (via their metadata APIs).
That said, it's super easy to set this up yourself on a cloud VM. You could easily bake the necessary bootstrapping into an AMI. Alternatively, you could use a startup script. One of the things we're working on is streamlining this experience. For now, here's a gist demonstrating the concept (it assumes you already have `step-ca` running):
https://gist.github.com/mmalone/a5980799c9c6ec9d6530372d5b60...
Most of the work there applies to other tools but obviously you'd use a different client instead of `step` to get the certs.
> How will this interact with other features like using a pkcs mechanism for a token key?
Oh hrm. This is a good question but I'm not sure, mostly because I haven't put SSH keys in a PKCS module myself. I know it's possible. You can implement the `ssh-agent` API pretty easily and back it by whatever you want. This is on our roadmap but we haven't done much investigation. It's possible that it just works already with standard `ssh` and `ssh-agent`s.
For MFA, you can still use PAM authentication with something like Duo if you want. Certificate authentication basically works exactly like public key authentication, but it manages public key distribution differently (using certificates).
That said, personally, I think the right place for MFA is during certificate issuance rather than on each SSH connection. That way you're not constantly asking people to MFA. This decision is somewhat subjective and depends on your risk profile / threat model.
> revoking
Generally the best practice for certificates is to issue them for a short enough duration that you're comfortable not doing revocation. I think a work day is pretty safe and reasonable, but some people may not be ok with that and choose something shorter. Of course you'd have to reauthn more frequently then to get a new certificate. A thought to ponder: for any reasonably sized infrastructure public key removal won't be instantaneous either.
That said, SSH does have cert revocation capabilities. You create a "revoke file" and configure it using a `RevokeFile <filename>` directive in your `sshd_config`. So it's managed on each host, which means there are no connectivity issues, but also means you have more of a management problem (on par with managing authorized_keys :/). For that reason, if you really need revocation, my recommendation is to do revocation checks in a bastion host. Then you only need to maintain the revocation list in one place.
> Also questions like what does an eaves dropper see in the clear and how does it differ from what they see with raw public key?
Great question. I should know the answer to this, but I don't. I just spent like 20 minutes perusing the specs at:
* https://www.openssh.com/specs.html
and it's not obvious just from skimming. Also couldn't find a definitive answer from the google machine. I'll need to read the specs more closely.
That said, it looks like pubkeys/certificates are exchanged during diffie-hellman key exchange, before a symmetric key has been negotiated. The implication is that this information is sent in cleartext. TLS1.2 does the same thing, but 1.3 keeps certificates confidential. So there's possibly an infrastructure enumeration vector here -- a passive attacker could see whatever info is in your certs (usernames, public keys, permissions). It looks like normal public key authentication would also transmit a lot of this same data, but maybe not the permissions. Again, I'd have to look at the spec more closely. I think for most people this is not a big risk and the profile looks pretty similar to pubkey authn, but I'm saying that without a complete understanding so caveat emptor.
> I find it insane that ssh ultimately ends up with a weaker trust model than a pretty average website in a typical install due to less certainty and clarity about what a normal certificate based configuration should look like.
Lol yes. Me too.
To be fair though the trust on first use model with raw pubkeys made SSH way easier to deploy, which was one of the things that allowed SSH to displace telnet and, from a security perspective, it was obviously a huge improvement over telnet. So I don't fault the SSH folks for this. But it's 2019 now so we can/should do better!
Perhaps the title could have been more diplomatic. Perhaps I was not sensitive enough to the feelings of folks who use non-certificate-based SSH authentication. But it’s just a title. I meant no harm. <3
Do you have any feedback on the content itself? :)
I don’t want to be antagonistic but that’s just not true. I’ve talked to a lot of people about this. Maybe 10% of people I’ve talked to know how to use ssh certs. These are technical people who are very smart and know what they are doing, and know what “grok” means. That’s why I wrote the post.
If you already knew the info in the post then cool! Sorry to waste your time.
If you already have Kerberos, fine. If not, certs are way easier to implement & more flexible.
Yea Kerberos is another option. If you have all the necessary pieces, that is... you’d need managed devices, an LDAP/AD setup, DNSSEC & SSHFP, PAM & various agents on servers.
Certificates offer all the same benefits, and a few more, with less work & fewer moving pieces. They’re also more flexible since you can control the auth flow to get a cert.
We don’t sell anything related to what’s in that post (actually we don’t sell anything right now). It’s all open source. We believe everyone deserves good PKI; that it’s an underutilized technology with bad tools. We have plans to make money off other stuff once that’s in place.
Still, this is hand-wavey and complicated and not ideal from a security perspective. It's a lot better than not having certificates at all, but it'd be even better if this gap were closed.
To close this gap we need some sort of enrollment process. The reason we didn't add this for MVP is it's kind of complicated. I think we'd need some policy at the CA that maps VM identities to the workload identities the VM is authorized to run. We need to figure out what would run this enrollment step to add the mappings (probably different for different stacks) and how that thing would authenticate to the CA.
We've also been a bit reluctant to add ad-hoc policy stuff to the CA because we have a generic policy solution that we've been working on. Once that's released it'll give us a much better foundation for this sort of stuff.
Finally, there are other ways to build a stronger enrollment mechanism today. We have a JWT-based one-time-token authentication mechanism[1] that you can use, where a "provisioner" (e.g., something in your CD pipeline like Puppet or Kubernetes) issues a one-time token for a workload to get a certificate from `step-ca`. In this flow the JWT contains the workload's identity, so whatever issues the JWT controls certificate enrollment. This flow has pretty much the same characteristics as an IID+enrollment flow.
Finally, we have ACME support coming soon (next week, actually). So that'll be another option if you want a stronger binding to an instance's FQDN.
Hope this makes sense. Happy to answer any additional questions!
[1] https://smallstep.com/docs/design-document/#jwk-provisioner
Yea I have a pretty good idea of what’s involved. None of that is impossible to do. Time consuming? Sure. Impossible? Far from it. The root programs have documented processes to get included and will (eventually) accept a legit new CA that’s passed WebTrust audit and has good practices (afaik). Hardest thing is finding someone to cross-sign to accelerate the process. That’s why I asked jaas if he would do so :P.
I run a company that does public key infrastructure stuff (smallstep.com) so I have the software and some of the people. I think I could track down some hosting and a couple operators from a partner or two. What’s left seems like maybe a few hundred thousand a year for audits and HSMs, mostly. Pretty sure I could scrape that together too.
Idunno though. Mostly it’s Friday night and this is a fun thought experiment.
Good thought on buying an existing CA. Are any currently up for sale, I wonder? If anyone who reads this knows, plz email me (mike at smallstep).
Also, timeline. It took Let’s Encrypt a couple years to get in root programs and have good penetration, right? Would it be faster or slower now?
If someone was willing to undergo this ordeal and could secure funding so it wouldn’t split the philanthropic community and affect LE, would LE be willing to cross-sign (once they’re in good standing) to get them off the ground, the way IdenTrust did for the ISRG roots?
Asking for a friend ;)
Certs have "TLS Client Authentication" key use set by default.
I can see pros/cons either way, just thought you might be interested in this feedback.
To be clear, all of the evidence shows that LE uses this power only for good. But that’s the threat. 30% market share is probably fine. 90% or 100% would be scary.
That said, even AWS's crappy implementation is super useful, and really the only good way to do this (that I know of?). We've tried to mitigate this risk somewhat by making tokens single use. I'd like to also add a way to send a token to `step-ca` to say "this server doesn't need a certificate" that basically marks the instance as "used" without issuing anything. If everything that uses IIDs did this, and you ran through all of your services at startup and either authenticated or said "I never need to use this service", it would provide some protection.
Still, to your point, AWS should fix their shit.
I'm probably pressing my luck promoting here but if you do a bunch of cert related stuff check out our `step certificate` command group at https://smallstep.com/docs/cli/certificate/#commands -- it does a bunch of cool stuff like dumping x509 as JSON and extracting public keys and linting certs and it's way easier to use than openssl. Might be useful in your scripts.
Do https://hub.helm.sh/charts/smallstep/step-certificates & https://hub.docker.com/r/smallstep/step-ca tick this box for you?
> Looking at the documentation it seems that you have to manually enter the password when you start up step-ca... environment... kubernetes secrets
There are a couple options here. You're only prompted for a password if your intermediate signing key (at `$(step path)/secrets/intermediate_ca_key` by default) is encrypted with a password. You can create your own signing key (e.g., with `step certificate create`) or remove the password (e.g., with `step crypto change-pass --no-password`) and you won't be prompted for one.
The `step-ca` binary also takes a `--password-file` flag. We chose to take this from file vs. environment or directly as a flag because files are the easiest thing to secure in general... and we're all about misuse resistance. If you really want to use a string or an environment variable you can use bash process substitution (e.g., `--password-file <(echo "pass")` or `--password-file <(echo $PASS)`).
We're also working on PKCS#11 HSM support and will be supporting cloud HSM/KMSs on all the popular clouds, which is probably the best and most secure option for most people.
> There's also the issue of backing up your CA secrets, e.g. if your step-ca process dies and you want to restart it somewhere else.
Yea this is super tricky. I think HSMs are probably the right answer. As it stands, the password protection on the signing keys mean it's somewhat ok (if not best practice) to backup the keys. Really though, the whole system is designed to make intermediate and even root rotation fairly easy so there's that option, too.
This is an area where we're still improving and definitely appreciate feedback from anyone who has thoughts.
> Documentation on how to set this up in a high availability fashion
Yea good call. We need that.
For anyone who's reading this you should be aware that the CA just needs a root and intermediate cert in `$(step path)/certs` and an intermediate key in `$(step path)/secrets`. You don't have to create these keys with `step ca init`. You can use the lower level `step certificate create` command group to create a second intermediate that hangs off an existing root, for example.
> I do like the relative simplicity of this compared to all the other CA solutions out there. Good luck and thanks for the work.
Thanks so much! <3.
Edit:
If you're doing k8s stuff also check out:
- https://github.com/smallstep/autocert
- https://github.com/smallstep/step-issuer
If you're using Envoy/Istio we're also iterating on this:
If you ever have a reason to check out the `step` / `step-ca` toolchain I'd love to chat about the differences you see. Message me here or shoot me an email (mike at smallstep).
For what it's worth, I'm pretty sure even Let's Encrypt wants to see competitors to Let's Encrypt.