If you’re not using SSH certificates you’re doing SSH wrong (2019)
smallstep.com
smallstep.com
SSH keys make sense. But certificates? Is this OIDC, SAML, what? Is it unreasonable to request better and deeper "how to do {new security thing}" when PKI is a new acronym to someone? Where can I point my data science managers so they can understand the need and how to implement measures to have security on PII-laden dashboards? As so on.
Security writing in "engagement" obsessed media yields lots of people screaming "FIRE!" whenever seeing something even theoretically flammable, and bandwagoneers— already imagining their hair is on fire— lambasting everyone not immediately evacuating for being careless about fire safety. It reminds me of politicians being 'tough on crime'— they reflexively jump at opportunities to tighten the screws regardless of its necessity or efficacy. It's an emotional response involving self-image, peer pressure, and fashion rather than rational cost benefit analysis.
Perfect is the enemy of good. Attacking every theoretical threat like an international bank's network admin yields no practical benefit for most. Not nobody but most. If this TLA is new to me, there will be another new one that people will lambast me for not knowing in a couple of years, max.
For me, this problem was a better fit for the Wizard of Oz than a security education resource— what I really needed was the right frame of mind rather than learning the implementation details of every incremental certificate authority update.
I evaluate my attack surfaces and reduce them if I can, evaluate the real importance of keeping what I'm protecting secret, implement standard precautions and architecture to mitigate those risks, pay attention to the systems, pay attention to new vulnerabilities, and re-evaluate upon changes. The process is technology-agnostic and only requires you deep dive into the stuff you need to know without feeling like you need a new certification ever 6 months to run your company's CalDAV server.
https://arstechnica.com/information-technology/2017/07/how-i...
What is the cost (in time and effort and manpower and complexity) to implement? What is the cost to maintain? What is the cost to manage, when you are adding and removing people often? What are the failure scenarios, when any one server that needs to manage things starts to become a liability for disaster recovery and redundancy purposes?
Sometimes the destination is clearly better than your current place, but the road to get there has a cost all its own that makes traveling it the non-optimal choice.
It's very easy for 1-3 admins to decide to implement something over 10-30 servers and keep themselves up to date and with the right access and knowledge to manage and maintain it. It's quite another thing when you're talking about hundreds of servers and you've implemented clear delineations about access and you have 10-20 admins ranging from junior to expert with associated levels of access to servers and tools, and the fleet of servers has evolved over years (or multiple decades, in some cases). Applying changes over that type of system can be complex and error prone and when it affects your ability to actually access and maintain the systems in question, it can be very hard to reason about the problems until you start encountering them. Change comes with risk, and risk assessment of technology becomes a large part of the planning requirements.
Excellent. This is often ignored by the security obsessed, those people yelling FIRE! as you say.
Securing access to my cloud-hosted cat photos does not demand the same energy as securing ICBM launch codes.
At work, we develop Teleport (https://goteleport.com/) to provide a secure access solution that is also easy to use and hard to get wrong. (Note: you cannot truly have "hard to use" and "secure" access, because people will always develop "backdoors" that are easier to use but not secure.)
If you are interested in some accessible writing about security check out: https://goteleport.com/blog/
On SAML: https://goteleport.com/blog/how-saml-authentication-works/
On OIDC: https://goteleport.com/blog/how-oidc-authentication-works/
I can recommend the YouTube channel too: https://www.youtube.com/channel/UCmtTJaeEKYxCjfNGiijOyJw
With that said, the company really needs to improve its interview process--my experience was downright terrible, and Glassdoor shows that other people had a similar experience
It really put me off. I’m not dead set on developing in any given language. I like rust and have been working with it for a while but that isn’t a deal breaker for me. The thing is that if our introduction starts off with dishonesty I don't have any reason to expect it to get better from there. What will they mislead me about after I’m hired?
https://goteleport.com/blog/coding-challenge/
We are also trying to be as transparent as possible with our challenges being open source:
https://github.com/gravitational/careers/tree/main/challenge...
and requirements being published here:
https://github.com/gravitational/careers/blob/main/levels.pd...
I am sorry to hear that you had bad experience. Our interview process is a trade-off and has one big downside - it may take more time and efforts compared to classic interviews. It could also feel disappointing if the team does not vote in favor of the candidate's application.
However, if there was something else wrong with your experience and you are willing to share, please send me an email to sasha@goteleport.com.
Devs (who arnt desperate for richers) look at your company and think, how cr*p would it be to work there? where are the indicators?
Personally I became interested in working for Teleport in large measure because the interview process tested my practical skills, rather than having me pull leetcode trivia out of my ass. I haven’t regretted my decision whatsoever, all of my engineering teammates here that I’ve worked directly with are very responsible and competent and the company appears to be growing mostly in the right directions.
* Error handling and code structure - whether the code processes errors well and has a clear and modular structure or crashes on invalid inputs, or the code works, but is all in one function.
* Communication - whether all PR comments have been acknowledged during the code review process and fixed.
Others, like whether the code uses good setup of HTTPS and has authn are more clear.
However, you have a good point. I will chat to the team and see if we can reduce the amount of things that are subject to personal interpretation and see if we can replace them with auto checks going forward.
Here's what I think it boils down to: working on a codebase with your coworkers is (or at least certainly should be) an inherently collaborative process. On the other hand, a job interview is, in a sense, inherently antagonistic. No matter what shape the interview takes, these people aren't your friends, they aren't your coworkers, they are gatekeepers.
I already have a job as a programmer. At work, I can push back on my coworkers and debate the merits of various designs until we all reach a consensus. But with the Teleport interview, there's an inherent power imbalance that makes that impossible: "I'd really like to argue about this, because I don't think I agree, but I'm afraid that will decrease the chances of them hiring me."
And the only people who are in a position to change this process are the ones who have already gotten through it successfully.
1) You’re assuming that a good faith argument would decrease the chances of us hiring you, but for the most part that isn’t the case. We’re an engineering company building a complex security product — the only way that can be done well is via a culture that’s perennially open to criticism, debate, and going with the better argument. In my tenure at Teleport, I’ve never experienced explicit or implicit punishment for voicing my opinion, even when it contradicted a more senior engineer’s opinion. The argument has always been evaluated on its merits and the correct option taken. An interviewee making a good argument and proving an interviewer wrong should, and based on my experience would, increase your chances of being hired.
2) I can imagine you retorting that even if that’s truly the case at Teleport, there’s no way you could know that beforehand, and due to the “antagonistic” nature of us being the “gatekeepers”, you’re forced to assume the worst. But if your goal is to work in a collaborative environment where criticism and debate is tolerated, then your implicit strategy makes no sense. If Teleport is that type of place you’d like to work, then pushback in the interview process will be well received; if it isn’t, then you won’t even get an offer. So you have nothing to lose by giving your true opinion, but if you assume the worst and self censor in an attempt to brown nose the hiring team, you risk ending up in a shitty work environment that you were hoping to avoid.
We are hiring and we have a non-terrible interview process (and amazing culture)!
I agree, our enterprise product is quite expensive. Let me explain why:
* We are going through several security audits by third party agencies several times per year. We are trying to hire the best security agencies to audit our code and it is quite expensive.
* We are recruiting globally and try to place our comp at 90th+ percentile of the compensation as listed in opencomp.com and other sources we have access to.
* Our sales process also takes time, and the sales team employs sales engineers, sales and customer success specialists to assist with deployments of such a critical piece of the infrastructure.
* For all our employees we have wellness benefits for home office improvement, personal development, healthcare packages.
All of these factors above add up and we charge a lot for building a quality security product supported 24/7 across the globe.
However, this might not work for everyone, and we have a completely free and open source version that people can use without ever talking to our sales team:
You don't get support and some other things (see: https://goteleport.com/docs/enterprise/introduction/), but this is not a "demo" version where you cannot do actual work.
Kind of crazy indeed.
All competitors i can think of are also expensive.
Seeing an "enterprise, call for a quote" type tier makes me assume it's going to be too expensive for agency securing 10-20 servers.
For our hosted product, you're looking at $30-$60/month.
I sign SSH certificates for all my keypairs on my client devices, principal is set to my unix username, expiry is some weeks or months.
The servers have my CA set via TrustedUserCAKeys in sshd_config (see manpages). SSH into root is forbidden per default, i SSH into an account with my principal name and then sudo or doas.
My gain in all of this: I have n clients and m servers. Instead of having to maintain all keys for all clients on all servers, i now only need to maintain the certificate on each client individually. If i loose or forget a client, its certificate runs out and becomes invalidated.
Compromises happen in seconds - milliseconds, and once they do they will establish persistence. Expiry systems do not and have never been protection against compromise. They're an auxiliary to revocation systems to let you keep revocation lists manageable.
If you don't have revocation lists, or your number of changes is small, you should go ahead and just set your credential expiries to whatever you want - infinity, 100 years, whatever - it won't make the slightest bit of difference.
Particularly in the case when they're protecting sudo user credentials, they're no defense at all.
I will argue though that the use of a short expiry produces slightly better protection than no expiry at all. If an employee leaves the company (with no CRL in place) and their certs expire in 16 hours, then unless their credentials are stolen in that timeframe your systems are still safe.
Likewise, if a CRL is in place and credentials are stolen without you being aware of it, the expiry still provides a form of buffer if the stolen credentials end up being used after the cert expires. In this case the expiry would trigger before you realised that credentials were stolen and updated the CRL. Now yes compromises can happen in seconds, but that's not in every single case.
That being said I definitely agree that the expiry is not a subsitute to a CRL and any certificate system should have revocation systems in place. In the end you really should have both a CRL and expiry date if possible.
And its actually a separate thing since it operates largely independently from the CA.
I have one in place. Used it once to terminate access for someone.
The infrastructure I built for access control using SSH certs used it. I know it works because I tested for it specifically.
> Yeah the lack of mentioning a CRL at all really stood out when reading this. I actually didn't know about SSH certificates until I saw this article (I always assumed that SSH did not support this), but do run my own CA and authentication for internal web services, EAP-TLS, and VPN. The CRL is your first line of defense in the sense that it blocks the use of that credential instantly when it is revoked.
This sounds like he/she is running an x509 CA. He/she is generating certs for various use-cases.
It is possible to use x509 certs with SSH of course, and so he/she could leverage his/her pre-existing CA for that function.
Given above context CRL is completely accurate. And, KRL is not.
In an organizational context, many organizations are not going to jump to creating a novel CA type (SSH CA) when in fact regular x509 CAs are well known and the basis for much security, and many in regulated industries are using them already.
Additionally, given that he/she is running an x509 CA, telling someone with that experience to study the fundamentals is not very polite. It assumes the author of the comment is not educated, but the very description of his/her use-cases are not simplistic ones.
Engineering is all about tradeoffs after all.
This is how this type of authentication works, and the article did not address the important case of wanting to revoke a user's credentials.
This is for symmetric encryption, and for asymmetric the equivalent is ~1024-bits, so padding it up to 2048-bits is generally the "minimum" for RSA, and some of that math is advancing too so bumping it to 4096-bits isn't a bad idea. If you want to be quantum proof, RSA will be broken so moving to something else like EC is nice. AES would be halved O(sqrt(N)), so AES-128 becomes the equivalent of AES-64, so if you want to be quantum proof there you need to jump up to AES-256 (unless you are using XTS/tweak mode, in which case AES-512). Keep in mind quantum also is not exactly short term practical to accomplish at the moment.
You can use whatever technology to accomplish that complexity, be it passwords, SSH keys, or SSH certs. Anything else is just technology architecture noise. Passwords absolutely can clear the O(2^N) > 80 bit threshold. It's just about bytes, and how you store them.
Nobody is going to be brute forcing a sufficiently complex password over the network anytime soon unless it isn't actually random but some default password that looks random.
Just look at the title of this post: "If you're not using SSH certificates you're doing SSH wrong". It's just completely devoid of environment issues, user issues, datacenter issues, and reeks of elitism. There is no "one true way" despite people's insistence that they are the arbiters of truth. I keep reading here about "you should just use serial over network instead of SSH!" but fail to read about how those serial over network connections are usually less secure than SSH itself.
Best practices guides have gone off the rails. They are generally good guidelines, but you have to make sure you are taking into account your own environment and user needs and take them with a grain of salt. Learn for yourself, and read raw facts from real cryptographers and people in the field. Don't take best practices guides as absolute truth, but learn from them.
How does one become a security professional? Maybe not with one of those "become a security professional in 30 minutes" packages then start a blog about how everyone isn't conforming to their tiny worldview. No matter what it'll take >10 years with actual experience, just like any profession. One has to start from the bottom and make their way up. Most environments are too complicated for any "one size fits all" solution:
EDIT: Further discussion on this here is interesting. The top comments go all in on SSH certificates, then down the line people start questioning why passwords are bad in the same ways. A lot of the "SSL certificate" push theorized here from their perspective seems to come from VPN providers that need it from lesser skilled clients/users (think, people who bought VPNs off YouTube video recommendations):
https://arstechnica.com/information-technology/2022/02/after...
I always try to assume breach in my thought processes, but I recognize that this lead to overengineered solutions because sometimes the mitigation is not worth the cost.
> Just look at the title of this post: "If you're not using SSH certificates you're doing SSH wrong". It's just completely devoid of environment issues, user issues, datacenter issues, and reeks of elitism.
I think this is an excellent point you make. There are a few different ways to use SSH securely and I probably lean a little towards the x509 and other alternatives, given the established base of x509 within my industry.
I don't use SSH certificates at work because they really don't make sense for me when I am using a strong credential already (HSMs)
> There is no "one true way" despite people's insistence that they are the arbiters of truth. I keep reading here about "you should just use serial over network instead of SSH!" but fail to read about how those serial over network connections are usually less secure than SSH itself. Best practices guides have gone off the rails. They are generally good guidelines, but you have to make sure you are taking into account your own environment and user needs and take them with a grain of salt. Learn for yourself, and read raw facts from real cryptographers and people in the field. Don't take best practices guides as absolute truth, but learn from them.
These are some other seasoned points you make.
I like to think about "Security Objectives". In most cases I am concerned about is something secure from a confidentiality, or integrity perspective. But, since I also deal with an ICS/SCADA community, their context is completely driven by "Availability as Paramount", defined performance within an acceptable range being next, and only after that, does the other objectives come into play.
However, given the varying use-cases of machine, mobile, app, connectivity basis or lack thereof (internet, transient, air-gap, etc) and the limitations of each, sometimes a smorgasboard of solutions are needed to satisfy within constraints.
> How does one become a security professional? Maybe not with one of those "become a security professional in 30 minutes" packages then start a blog about how everyone isn't conforming to their tiny worldview. No matter what it'll take >10 years with actual experience, just like any profession. One has to start from the bottom and make their way up. Most environments are too complicated for any "one size fits all" solution:
Appreciate the words of wisdom.
I view security as having much in common with other rapidly evolving fields of expertise. The generalists becoming specialists, are now becoming sub-specialties, adding fellowships, etc. When I was a young force-sensitive had the good fortune to fall in with the right community in which to collaborate.
My opinion is that many of the security communities are among the most welcome, diverse, and inviting folks around.
"DevSec SSH Baseline" ssh_spec.rb, sshd_spec.rb https://github.com/dev-sec/ssh-baseline/blob/master/controls...
"SLIP-0039: Shamir's Secret-Sharing for Mnemonic Codes" https://github.com/satoshilabs/slips/blob/master/slip-0039.m...
> Shamir's secret-sharing provides a better mechanism for backing up secrets by distributing custodianship among a number of trusted parties in a manner that can prevent loss even if one or a few of those parties become compromised.
> However, the lack of SSS standardization to date presents a risk of being unable to perform secret recovery in the future should the tooling change. Therefore, we propose standardizing SSS so that SLIP-0039 compatible implementations will be interoperable.
I am equally mystified... Never understood how involving a possibly malicious third party can make communication more trustworthy.
But then again, I was also sure when I first heard about it that public key cryptography was obviously impossible. You just could not have secret communication when everything is on the open! Is there any simple explanation that we ignorant people can read about certificates to get an "aha! insight" moment? For the case of public key cryptography, the moment where everything snapped together was when I read the mathematical description of the Diffie-Helman key exchange [0].
I'm not interested in how to do certificates with ssh, but on what problem do certificates solve, exactly.
[0] https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exc...
(1) You want all of your authentication to route through a single point of control where you can enforce group-based access control, MFA, onboarding/offboarding, and audit logging.
(2) You want the actual secrets that allow you access to an SSH server not to live for a long time on anyone's laptop, because it is effectively impossible to ensure that, on a sufficiently large engineering team, nobody's laptop will ever get compromised; there's just too many of them, and developers do weird shit so the machines can't be ruthlessly locked down. You want people to have SSH login secrets for exactly as long as they need them for a specific server, and no longer.
Certificates solve the problem of having dynamic access control to SSH servers without having some weird system that is constantly replacing authorized_keys on all your servers; instead, there's a single root of trust on all the SSH servers (the CA public key) and a single place that mints valid certificates that enforces all the stuff I mentioned in (1) above.
It's worth knowing here that SSH certificates are nothing like X.509 certs; they're far simpler, and you could bang out an implementation of them yourself in a couple hours if you wanted.
On the other hand, they are useful for large organizations, with needs for differentiated access rights and management rights.
The centralized control over the certification authority allows the delegation of restricted rights to other levels of network administration.
Everybody knows the public key associated with that private key, so you can verify that the private key did sign this list of attributes.
An ssh keypair is an actual public/private keypair, but a certificate is just signed and encoded (but not encrypted) formatted data.
If an ssh daemon has knowledge of a public key used to sign a cert, and has been instructed to trust that cert, and all the dates are good, then the ssh daemon can accept that cert as proof of identity and allow a login.
But why would you want to do that? What problem does it solve? Just that you can connect without having a private key yourself? This doesn't sound very safe.
I have n clients, m servers.
On clients, i sign the lokal keypair with the CA key and log-in via certificate. The client-side certificate basically replaces the line in the server-side authorized_keys. The editing stays locally.
On servers, i register the CA key as "Certificates signed by this keypair are trustable", the authorized_keys file stays empty. No further editing required.
During normal daywork, the CA key sits unused and can be shut away.
Key Advantage: I don't need to edit anything on the countless servers anymore.
I think people really get into trouble with SSH certificates trying to reason about the properties of certificates versus SSH keys versus passwords. The format isn't the point; making the endpoint keys dynamic is. If you built a secure messaging system that propagated one-time-use SSH keys, it would address the same problem. Nobody will, because certificates are easier and already work, but you could.
If you connect to a ssh-server for the first time, ssh will give you a warning and let you know that you have to verify the fingerprint of the host key.
This becomes annoying when you connect to many different servers and I would not trust everyone (including me) to do this check correctly every single time.
SSH certificates solve this by having the ssh-host-key be signed in a way that your ssh-client can verify and you only have to add a key-signing-key to you known_hosts once.
Now you have to sign the ssh-host-key but you only have to do it once per server as opposed to having each user having to do it locally on every first connect.
It would be nice if there was a dummies guide but I'm not really aware of one. Doesn't help that most of "how to PKI" on the web amounts to a bunch of unexplained cryptic openssl CLI incantations.
I started working in this space a year ago (I'm on a project deploying zero trust networking at a large company) and these books have been invaluable.
Main issue is that most people do not even grasp the concept of a certificate (ie. binding of public key to some additional information that is signed by some other entity).
So it isn't only the technicalities of asynchronous encryption, there is also specific behavior of applications that use certificates to prove identities.
It's nice in that it will list out a bunch of available encryption algorithms or hash algorithms, but at the end of the chapter say "Just use this one, it's considered safe right now." i.e. AES256 and SHA256.
Unfortunately, it mostly avoids the practical steps of web security, like its not going to print out the command to type in to your shell to generate an SSL signing certificate. So I wouldn't recommend it if you're looking for an immediately practical book to help you secure your web server. But it orients you to the landscape so you have a general idea of what you're trying to achieve, and can google yourself the rest of the way there.
Also, many of the chapters are available to read for free - read author's text under the cover photo.
I'll look into this and perhaps supplement with some good tutorials for my developers and data scientists. I appreciate your input!
SaaS like Smallstep and Teleport are trying to middleman and monetize what is actually a simple process that more developers should be comfortable implementing themselves.
This isn't "rolling your own crypto", this is standard SSH key stuff plus a bit more nuance and LOC to make things more secure.
When you pay for those services, you are essentially paying for a wrapped SSH command + dead-simple web app + someone to be your CA (read: store the resulting files of `ssh-keygen`, hopefully securely). And all the potential headaches of relying on yet another SaaS.
Plenty of developers are comfortable writing a script, a simple web app, and securely storing a file on the webserver. This is all that is required to build SSH cert support into your internal apps/tools, plus an afternoon understanding how CAs work (in short, CA private key can sign any SSH public key, then that SSH public key can be validated by anyone holding the CA public key, no TOFU required).
I haven't read it yet, so I'm posting i hope of someone else giving a quick review.
If you are having 10-50 servers and 5-10 people working on those - SSH keys are definitely good enough, it might be a bit of hassle to manage keys but quite OK.
If you go into large corporation area with more than 100 servers and more than 50 tech people that need to login to those servers you probably would already found out that there are other options and you probably have to run your internal CA (certificate authority).
If your org grows you would probably have CTO and other technical people who will have experience knowledge to implement things differently.
[1]: https://github.com/FiloSottile/age
[2]: https://github.com/ryantm/agenix
[3]: https://calebhearth.com/sign-git-with-ssh
[4]: https://www.man7.org/linux/man-pages/man1/ssh-keygen.1.html
If you are constantly regenerating uncompromised SSH keys, you are probably doing something wrong.
The GP is misleading in this way.
Yup, there's no real reason to generate a new SSH key pair each and every time you want to get a short lived certificate. The SSO or CA management system (like Vault) is responsible for verifying your identity.
It might make sense to regenerate the private key on each certificate renewal if the private keys are kept unencrypted, as they often are in the X.509 scenario. If the keys are encrypted and if your web server manages to get the password to decrypt that key each and every time it serves a request, then yeah, I don't see the point of regenerating the private key on certificate renewal.
I went through the article again. Essentially, rekeying makes sense if the private keys in question, whether they are host private keys or client private keys, are kept unencrypted on disk. Host private keys typically are, so it might make sense to rekey host private keys. However, if your user private key is kept encrypted on disk, as it should be, there isn't really a good reason to rekey.
The step tool seems to abstract that process and it also generates a new key pair on each login but that keypair never even touches the disk, according to the article. This makes sense assuming the step tool generates a key pair and doesn't encrypt the private key. In that case, yes, rotating/regenerating the client keypair on each login make sense.
> Signing certificates with your existing keys kind of make the whole point moot.
You mean getting signed certificates from a SSH CA with an existing client public key? Why does it make the whole point moot? The SSO is responsible for verifying your identity. Rather, it seems pointless to generate a new SSH keypair each and everytime you want to get a certificate and login to a machine. You can certainly do it if you want but I don't see the point. You can keep using your existing SSH public keys to get short lived certificates and login to a machine, do you job, logout, and repeat the process with the same keys.
One of the primary benefits of the described setup is that there would only be a single long term key. It would be managed more securely because it won't be lying around on each user's personal machines.
That certificate would only be issued after you've gone through SSO or some other certificate management process and it would have a defined expiry period, maybe even as short as a few minutes. It is a meaningful change.
When using key based authentication, as long as you had your public key in the authorized_keys file on the server, you would be able to login without issues, even if your keypair is compromised. With certificates, even if your keypair is compromised, you wouldn't be able to login to that server because you'd have to compromise the SSO/MFA authentication step as well, which adds another layer of meaningful security in the process.
Different keys for uhh, different complex application use cases.
To really secure your SSH server, do these three things:
1. Setup a trusted authority
2. Use ssh certificates
3. With ssh certs, you can easily add MFA for logins expire until you reauth.
Now that we've established this, here are the gory details.
Lorem ipsum, lorem ipsum, lorem ipsum.
This drives me a bit nuts about security people in the age of cloudscale. They assume you don't mind MFA'ing every hour and are manually doing logins and accesses for everything. Yeah, uh, I need to script orchestration on several hundred machines at once, and orchestrate/access on the scale of hours or even days for some things like "Big Data" backups or restores.
If certificates are anything like SSL certs and the horrorshow cli tools / options / management involved in those, no thanks. I'd rather have an automated sshkey switchover, or for stateless just routinely cycle the infrastructure with new keys.
It's been a long time tenet of security that you want an open algorithm that gets broadly and publicly challenged so you know its secure. Well, in the age of the state actors, this might not be the whole truth.
I think layering some klugy not-invented-here obfuscation atop the more battletested methods is a useful and important deterrent/delay. Sure someone will figure it out, but they have to TRY HARD. A lot of the institutional attacks seem to be based on human attacks on standardized systems, which HAVE to allow human access and vectors.
So in AWS land, secrets manager is secure unless you get the permissions. Then you have everything. But if each of those secrets has some whacko obfuscation for the various apps, then that is a big slowdown to the human attack vectors.
And the state actors? Well, even they have budgets. They'll probably move on to easier targets. If a state actor is motivated at targeting your company in particular, well, given that they'll have malware in the firmware of your hard drives and motherboards and the like, you're probably helpless.
Finally, what really bothers me about most security is that it leaves one of the most important "canaries in the coal mines" aspects of security: honeypots. Sure do the diligence on securing the access, but how about some turnkey approaches for setting up honeypots to detect when people are poking around? Honeypots are perfect for that, because the devs only care about the stuff they are working on. The intruders are doing the scanning.
Over the last 6 years I've worked for three different companies that all universally disabled ssh and we never had troubles running management scripts or tooling.
Disposable API servers you can eliminate ssh access and the like. Containers generally don't run sshd, but they do still often have kubectl/dockerrun if you absolutely need to.
SSM sucks on a certain level because the output is capped at ?1MB? I think and you need to poll S3 to use it.
Salt daemon polls fine.
As you kind of alluded to, well, you can do SSM to "port knock" or simply do a change to a secgrp to flip on the ssh access, and then flip it off afterward.
I find Salt/Ansible/k8s too big, you can't "step through" to debug your orchestrations. I would categorize them as "heavyweight". SSH is a good substrate for everything else, including adhoc stuff.
Anything that can be run off a laptop can be run off an admin server. Remotely debugged. I love it. With the heavyweight stuff there is too much trial and error: try recipe, it craps, guess what's wrong, it craps, guess again, it craps.
The stuff I use actually doesn't require SSH. As long as you can deliver a command and get the output, I can use SSH, SSM (with the crappy limitations but its GREAT for stuff in China), kubectl, salt, dockerrun, teleport (until the token runs out), or combo ssh-to-bastion then do other stuff.
Some deployments make the daemon approach (that phones home) difficult. Such as management in a corporate network. It's easy to configure AWS and the like to accept requests from well known corporate gateways. It's not as easy to make them from the outside the corporate network in. And even when that's doable, different cloud providers and regions make it difficult. You end up having a bunch of chef (or similar) servers scattered around.
SSM sounds like an advanced port knock. Or you could toggle the security group port access, or keep the bastion down and spin it up if you need it.
Or OpenCanary if you can't afford $arm&leg?
Which says something... but then again if the security group is doing its job with setting up canaries, why would peon dev like me even be awares?
a lot of blah-blah that makes my eyes glaze/lose focus before getting to the core/overview.
~15 years ago, when the company I'm working for first implemented PKI, I needed something like 100 hours of "help" from local security engineers and the external software-vendor's programmers to understand how that worked and what the SW was supposed to do. After that, explaining to colleagues at least on a high level how that works became a matter of minutes.
To be fair towards myself, even the local scrty eng gurus (relaxed, as they themselves didn't have to "deliver" anything) and the external vendor's programming gurus (hardcore, as nothing would be paid if the SW could not be implmemented) were often absolutely not understanding each other, so I ended up becoming their unofficial mediator/translator:
if I felt like an obvious question was not dared to be asked by one of the parties then I sacrificed my self-esteem to dare asking it, if "even I" did manage to understand some concept then both parties were supposed to get it as well and if not then at least I could act as gateway to explain/expand offline :P
It was an interesting time - not nice nor very bad (a little bit bad, as we had of course an implementation deadline), but at least I learned a lot, as well in the area of social skills :)
* Transcript-level audit trails for what people are actually doing on SSH sessions.
* Differential access to different groups of users to the same machines.
Tailscale and SSH CAs work together nicely: require membership in the right Tailscale group to talk to SSH at all, thus tying access to SSH to your (e.g.) Google login and MFA requirement, and use something like Teleport for the actual SSH login, to get the audit log, group access, and an additional authentication factor.
The fact you can ping the server shouldn't mean you are allowed to actually access it.
Seems more secure than handing over the security of your servers to a CA?
Managing the authenticated_keys file is trivial and I like simple file based solutions.
I find simpler is often more secure because its easier to understand.
I have had plenty of ssh servers on the Internet even, (on non-22 ports to prevent logs filling) and not had a problem yet AFAIK.
I have also been on call when certs expired gawd knows how many times.
Situation: A person's machine dies and takes the SSH private key with it
No Certs: They now need to get IT to distribute the public keys to a set of hosts, where the full list is possibly not known by any one person, so for the next week it will be "oh shit, I forgot to ask IT to put the pub key on server-12, so I'll open a ticket and not get any work done for the next few hours"
With Certs: They get IT to sign the new key.
If the team responsible for server security isn't able to identify all the servers they're responsible for, you've got far bigger problems than SSH certificates are going to solve.
Not sure I buy that, people take backups. No reason why a CA can't loose its private key if we're presuming backups are not being taken, and then everyone is affected.
You can also bake some basic rules/policies into the certificate, i.e. the above flow returns a signed certificate that is only valid for SSH'ing into hosts in the 'www' group (not the 'database' group.) So then operations people can add you to groups in some other place (LDAP, Exchange, whatever) and when they add you to the 'database' group, any future short-term SSH keys issued by the central authority allow you to now shell into the 'database' hosts. The hosts themselves can also have some automation to check these permissions are accurate when an SSH login request occurs. So you just file an IT ticket asking to shell into a host or group of hosts, they fiddle with some knobs somewhere, and a few minutes later you're done.
So the actual second part is more like "Get a new laptop and log back in through Okta" or whatever, and you have all the exact same permissions and whatnot you did before. It's easier for both the user and the administrators.
An immediate advantage of this are that keys are easy to immediately audit and revoke (because a central authority issues them so you have a trusted trail). But there are some more subtle advantages; one for example is that you make servers more homogenous and identical. They all just ship some specific sshd_config file, normally identical, rather than each server potentially having a different authorized_keys file; instead the central authority that issues keys is where the mapping of hosts/authorized users is, rather than each server having that knowledge "individually" in its own file. That's easier to understand, track, keep up to date (who has access where?) etc. You can just go modify a single global user and have the permissions flow downwards.
None of this really matters if you only have like 1 or 2 people doing everything, or it's your home network, but once you get above like, 5-10 people, or you have at least one sysadmin, it's actually really useful IMO. Also, to some extent, you can mix and match various parts of the above concepts (e.g. running code on every login request, to add extra authorization checks based on out-of-band information.) You have to decide what's appropriate for you. Read the ssh and sshd man pages and you'd be surprised at what you can do.
Source: I basically implemented my own SSH certificate authority infrastructure (server/client automation) "for fun."
Also, users tend to hate having to always relogin to SSO; maybe that's because the implementations have poor UX, and maybe there's no secure way around it.
One thing SSH certificates certainly have going for them is that they're actually easy to script and integrate with, and "piecewise" migrate to, in my experience, while using a flow you already are pretty familiar with. I personally didn't use any sort of LDAP or AD setup to back my design; you can implement a custom backend for all this pretty easily yourself. There's nothing inherently confusing about the concept of cryptographic certificate authorities or anything, anymore than public key cryptography itself. It's a relatively natural extension of the SSH design you know already, is my point. Again, the man page is worth reading to understand it all a bit better.
> Also, users tend to hate having to always relogin to SSO; maybe that's because the implementations have poor UX, and maybe there's no secure way around it.
Well, I'll be honest, people who tend to use SSH and would be impacted by this stuff tend to hate lots of things and not always for good reasons. Put another way, listening to developers or whatever about what they hate and what's actually good isn't something I would factor into something like this. SSO is mandatory for very good reasons at any reasonable scale (and by "reasonable" my opinion is you should have it in place at, like, 10+ people.)
Anyway, besides that. There's nothing in theory that prevents you from doing something specific like having the backend refresh the SSO token issued for your SSH certificates every time you log into some server, upto some given interval e.g. logging in at least once a day seems reasonable, but if you login every 5 minutes to a new set of hosts you can refresh the token.
In my case the flow was something like 'my-ssh-ca-wrapper ssh user@bar', which would ask you for a token. I would then get this token by visiting a little webpage I wrote, but in theory it could also just launch the browser itself with xdg-open with a direct link. I just use a password manager to fill out those "SSO" credentials. It isn't ideal or fully integrated but in practice it would only take a few seconds and it's similar enough to corporate SSO setups. But yes, polish is everything for those final few steps. The actual backbone is pretty straightforward, though.
A new machine should just have a new name. If one really wants to pretend that it's the old one, they'd better really copy it, including the keys. But even skipping that, sorting this out doesn't seem like a big deal (at least at a small scale; I suspect the article makes more sense in some scenarios than in others).
> Curiously, OpenSSH chooses to soft-fail with an easily bypassed prompt when the key isn’t known (TOFU), but hard-fails with a much scarier and harder to bypass error when there’s a mismatch.
Seems to me like a sensible behaviour for TOFU, not sure what's curious about it. Sounds like it implies that an unknown key is at least as bad as a different-than-known key, but that sounds wrong in context of TOFU.
> Once the user completes SSO, a bearer token (e.g., an OIDC identity token) is returned to the login utility. The utility generates a new key pair and requests a signed certificate from the CA, using the bearer token to authenticate and authorize the certificate request.
So the weakest point will likely be the SSO and the related infrastructure, instead of SSH and actual keys, and you'll probably depend on third-party services and/or custom/uncommon self-hosted infrastructure. Likely with a SPOF too. Doesn't sound good in general.
It probably does make sense in some organizations, but this particular setup doesn't seem to apply to all SSH uses, and to justify the title.
(Let's not pretend that "SSO" doesn't mean "let Google or Microsoft handle password storage for me".)
(1) Authentication bypass to your email service is already game-over for almost every serious company.
(2) Google (if that's what you're using) has better MFA and access control than what you're going to roll yourself.
(3) Having multiple sources of truth for authentication is a corpsec nightmare, and companies that don't have that invariably wind up accidentally persisting access for departed team members or contractors, and, worse, no single place to consult for a reliable catalog of who has access to what, which is why if you poll CISOs at large-ish tech companies, they'll universally tell you than one of the first 5 things they did when they took over was get SSO stood up.
(4) The "automated key distribution and revocation" system you roll yourself will be jankier and less safe than the certificate-based systems that already exist.
(5) Because that automated key distribution and revocation system does not in fact exist, what you're really saying is that you're going to live with developers having long-lived keys on their laptops.
If you don't trust Google, set up Shibboleth or something; the Google stuff is a sideshow. But the idea that you should manage SSH authentication separately from the rest of your authentication is pretty unserious. I spent about 4 years, recently, parachuting into dozens of mid-sized startups, all of them clueful, and except for the teams that had SSO-linked SSH access, SSH management was invariably a total nightmare. The "just manage SSH directly" approach is, empirically, a failed model.
> the rest of your authentication
What is "the rest of your authentication" in this context? Corporate email? As far as I know, SSH is the only real authentication possible here.
Traditional ssh-key auth is simple and reliable, it's not until you have a large, complex and diverse user base that you need something more. That's why the huge fang sites use it. Every org doesn't need to mimic fang.
I don't understand this obsession with enterprise-level security on home networks and hobby projects. If you think it's fun and educational to set up, then you're doing it for fun and education, not security. If you're doing it for security, you're basically setting up anti aircraft guns to do what a drone jammer could do with way less resources spent.
Good analogy.
SSH keys would make managing these few nodes a lot more complex that it is.
I implemented my own SSH certificate authority myself more or less, and while it's overkill for my own homelab-level stuff, I absolutely would never use anything else once I have more than like, 5 people logging into some set of machines. The benefits of centralized SSH access control that you can freely integrate (and pretty easily too, thanks to OpenSSH!) with your existing identity provider is really nice.
Sucks that Github and some other things force SSH keys which are just passwords except always saved to your disk so that anyone who steals your laptop gets access.
It adds insult to injury when you try to capitulate to this malarkey, generate a key in PuTTy's key generator, then Github whines that the default setting isn't overkill enough and you have to make a whole NEW key with some other setting. I miss the good old days.
This is the reason to encrypt ssh private keys with a passphrase. If the key is leaked it's still protected by the password.
It's a built-in feature of ssh. For an existing key downloaded from a cloud provider, use ssh-keygen -p to add/change the passphrase.
I don't keep an ssh key on disk though. I use my gpg key on my hardware security token, which gives you 3 attempts before you have to unblock it with a separate management password, which again you get 3 attempts at before the key is entirely locked.
ssh-agent will cache the passphrase in memory, which helps avoid needing to type in a long phrase repeatedly.
But it's worth saying that if any private key is leaked (passphrase or not), it's time to revoke it and generate a new one.
Having a passphrase in place raises the bar from "key leaked, 3rd party has access to everything" to "key leaked, 3rd party has to now attempt to crack the passphrase". It mitigates a very bad scenario and buys time.
Soylent Green is NOT people
Was one of my shorter pass phrases
SSH certificates are a solution to that problem.
You can secure access to Github (and other places) with hardware keys, e.g. from https://www.yubico.com/.
And not everyone saves it to disk. My ssh key is my gpg key. It's stored on a yubikey and can't ever leave it. If I do a `git pull` then my yubikey flashes and I have to tap it to allow that connection to happen. Steal my yubikey, well you can't unlock it. Hack my laptop and you can't tap the key.
Even for personal stuff, why would you want to use passwords? Keys are more secure AND more convenient. Sure you don't need certificates but I don't understand how keys are more 'nonsensical' than passwords.
Keys have more flexibility, you can use SSH Agent, you can do SSH agent forwarding, etc.
> except always saved to your disk so that anyone who steals your laptop gets access.
This is wrong. First of all, your laptop should have disk encryption. Always. I don't care what your threat model is, encrypt the disk. Second, SSH keys can (and SHOULD) have a passphrase.
I multiplied by cost per min for downtime in professional support and the cost of typing passwords was more than my yearly wage.
Keys are not just "passwords saved to disk". My private keys exist in hardware, on Yubikeys. They aren't on disk. The hardware requires authentication to access.
You're typing your passwords zillions of times. I log in to my system once, authenticate to my HSM once, and I can access many hosts via scripts and automated tools. This is impossible to do securely with password auth.
You're also training yourself to manually input the entirety of your authentication credential multiple times per day (or hour). This is bad practice, as anyone stealing it then has the keys (ha) to your kingdom (and they have way more opportunities to steal it!). Even if you just replace password auth with a password-protected key on disk, and don't use a password-caching agent that holds the decrypted key in ram (as would be typical), so that you're still typing your password each and every authentication, you've raised the bar substantially because someone would need to steal your encrypted key from disk in addition to obtaining your password.
Then there's the issue of cycling credentials, and the mental loads involved. I can cycle my keys without changing my workflow or having to type anything differently.
Passwords are not good authentication tools. Use actual cryptography.
further, this is even more convenient when paired with an ssh-agent that will securely hold your private key in memory and not allow anyone to export that key...you could dump the memory but that would require root access, which again should be password protected
As far as provisioning, maintaining a secure CA signing practice is a nightmare. It's K8S level of self-inflicted pain for a startup. If you're running at a larger scale and can dedicate a team to it, fine. If you're a dozen people trying to launch, getting the devops guy to run `ssh-copy-id` is not the challenge that this article makes it out to be. Nor is the slightly more automated Terraform script that installs and uninstalls authorized keys from servers.
I guess it's time to toss out my local (self-hosted) Userify setup that has been reliably working for years, where I can just instantly update my keys across all servers, and still log in even if some bad guys start DDoS'ing my Userify host, and just switch over to certs.
Oh, wait, now I see. This is a sales pitch for their web-based SSO app. If you don't use it, you're "doing SSH wrong". Good to know.
I quite agree: as long as it's not as easy as Let's Encrypt then it won't take off.
SSH certificates themselves are similar to x.509 but a lighter version. They are supported out of the box.
> Note that OpenSSH certificates are a different, and much simpler, format to the X.509 certificates used in ssl(8).
Where do you work with x.509 certificates when doing an SSH ca? I use SSH certificates productively and only ever used ssh-keygen commands.
x509 supports all of the things that ssh key file format does ... and also a lot more. ssh supports x.509 too so that makes it "easier". When you use `ssh` to connect, you'd specify -I and point to the private key and it will automatically try to find a certificate file whose name is identical to the identity file but with "-cert.pub" appended (see `man ssh_config`). There's another option to explicitly specify the key certificate file `CertificateFile` but there's no short-argument version so you have to use `-o CertificateFile [filename]` or add that to your `~/ssh/.config` file.
You'd need to use `openssl` command (or, of course, any openssl-like command line) to sign the your SSH key. And, of course, openssl doesn't understand ssh key files so you have to use `ssh-keygen` to convert the key to x509 format or else generate the key using `openssl` (but again, at least ssh understands x509 natively so the downside is having the much longer x509 text). And that's quite the arcane part: openssl cli is fucking awful. It doesn't follow normal command line conventions, doesn't have tab autocompletion, and its documentation is obscure/difficult to find, extremely terse, and difficult to even understand if you're not already explicitly familiar with exactly your inputs and outputs.
And then there's the whole process of getting your key file signed. At least that process is (in general) identical to having a web SSL key signed -- because the private key is actually identical but the only nominal difference is the format of the file that you normally think of using for it. But the workflow is different because the certificate doesn't get installed to somewhere that a webserver would want. I had ended up creating my own Certificate Authority to test with and that was yet another rabbit hole of anger management.
Thanks! That sounds quite condescending. I'd educated myself by reading about ssh keys in ssh and wondering how to use them.
Perhaps you should make your own blog post describing how to do it all using only ssh-keygen then.
Many folks are comfortable with jumping through the hoops (following instructions) WITHOUT understanding what's really happening.
Those of us who aren't comfortable with "just works" need to plow through a enormous amount of persnickety jargon and easily forgettable material to reach an understanding that gives us confidence.
For example, I want hostA and hostB to use the same CA. But some users should only have access to hostA but others should only have access to hostB. Others may have access to both.
Disclaimer: day work.
On the server side, set the AuthorizedPrincipals* for root to the list of allowed teams.
I currently use this method to store my SSH keys safely. But I don't know how this would work with certificates. If I have to store them in the computer instead of a hardware token it's a huge step back in security.
By the way what do home users use to set up a PKI? Scripting everything with OpenSSL is but very nice. It would be cool if there were an open source PKI platform ideally even with IDP built in. With a nice web interface and easy to install with docker. Never found one though.
Including for the CA key.
The toolchain is a real PITA though. Unstable, difficult to provision. Proprietary tools for card management. Not all platforms support PKCS modules. As far as I remember openssh on macOS was compiled with support for it. So I had to replace it with one from brew which is much harder these days. Mind you we're talking 5-6 years ago.
OpenPGP is really user-friendly. Nice config menu with gpg --card-edit . SSH agent functionality built into the gpg agent.
I kinda want to retain this level of comfort to be honest.
And of course once you do fido, you're back to the same issues around SSH keys that certificates are a solution to (as the article demonstrates). So moving to Fido is not a whole lot better than using SSH keys which work very well everywhere.
Also, another important point: I also use GPG a lot to encrypt files. It's great to use the same key (sometimes, for sensitive stuff I use a different OpenPGP card) and toolchain (always) for this. So I need it anyway, might as well use it for SSH authentication as well.
It works quite nicely, but the PIN has to be entered every time I auth to SSH. This may be a desirable feature to some, but I prefer the GPG way of the PIN being cached until the key is removed from the system.
(This could possibly be a shortcoming of KWallet, as it would pop up the dialog asking for the PIN, but checking the "remember this password" would achieve absolutely nothing, and besides, I wouldn't want to save it permanently, which that checkbox would otherwise do.)
I want GPG to ask for the pincode once, and the yubikey to require a physical touch for each authentication (including the first, obviously). This way it can't be automated by malware either (a pincode can be sniffed and replayed through the keyboard driver, the physical touch can't).
The PIN is great against attackers that find your key on the street. Not against a determined attacker that is already on your computer. For that the physical touch thing is a great solution (though the yubikey doesn't require it by default, you can easily turn it on).
Asking the pin upon first use and a touch every time is the perfect compromise between security and usability IMO. There's still some weakness around attackers with physical access but they are more easily mitigated.
My sshd is behind spiped + logging in requires me to physically tap my yubikey.
I understand the theoretical superiority to keys, but do we have some data per practically how many times key security actually failed someone?
Setting some sane security parameters for your SSH setup looks like a less jarring/drastic approach into securing SSH further[1]:
- Use keys.
- Allowing only strong cyphers.
- Remove weak primes.
> Keys are trusted permanently, so mistakes are fail-open.
But you can make the same mistake using certificates, by issuing a certificate with a large expiration date. AFAIR you _can_ revoke valid certs, but that involves making changes to every box running ssh, much like when revoking a public key.
I used to have a central location with an revocation list. All servers had a cronjob fetching it.
Instant revocation seems to be a giant hole in this article's argument which isn't addressed at all. If you fire somebody partway through the work day, end-of-day cert expiry is not good enough to prevent compromises.
I’ve actually started working on such an app recently, including a web portal, CA rotation, automated configuration distribution, etc. Still far from usable, but if you’re interested in contributing: https://github.com/Radiergummi/fides
Actually they do where I work, pretty much every time, and I didn't ask them to. Eyebrows will raise even higher if I forget to notify everyone I replaced a server at a domain causing a new signature (resulting in the scary "possible MITM attack" message). This is a good thing, but I should probably make it more efficient by publishing the fingerprints.
Although the article does point out this specific disadvantage of domain reuse with known_hosts. I can see why this solution could make things easier at larger scales.
When I last ran a fleet of servers, I published the known hosts files via git, and strongly suggested using that in new user documentation (you needed to setup ssh_config for our jumphost/bastion anyway, may as well link to the known host keys). There's a tool to generate the files that comes with openssh, iirc.
For anyone who is interested, I put together a little playground which can be spun up in Docker that allows you to play around with and learn how SSH CAs and Principals work:
However, I don't think openssh-server supports OCSP natively, so while you might be doing SSH right, you're doing certificates wrong.
Does the current OpenSSL have its own KRL?
Honestly what I would prefer is a better (and faster) version of monkeysphere [1]. That was honestly the most natural-feeling solution to SSH security.
(1) https://www.systutorials.com/docs/linux/man/1-monkeysphere/
For the SSH certificate to be accepted, the unix user must first be present on the system. As far as I can understand, FreeIPA(or similar LDAP systems) cannot be used in conjunction with SSH certs. Whereas SSH keys are supported by these systems.
Can anyone provide any insight/experience with this?
You can define principals when allowing a CA via authorized_keys, or you can configure allowed principals globally using sshd_config directives like AuthorizedPrincipals* .
It was actually quite an elegant setup. You would still need to setup a CA for generating local certificates for TLS connections to LDAPS, but the auth was handled all in the LDAP server.
I think the main downside would be trying to have the authentication overhead on a single server (the ldap server) when you are dealing with many hosts. Over a handful of systems, it’s great. But it doesn’t scale when you’re taking thousands of hosts (or cloud vms that spin up/down).
SSH keys seem great at first, 4KiB ~ 8 KiB public/private key-pairs are tremendously more secure than something like an 10-character password. The math checks out at an academic level, but the implementation has a glaring flaw. One cannot easily ensure private keys are themselves protected by a 10-character password unlock. Put another way, people using private keys NOT protected by a secret pass/phrase are super vulnerable to compromise. For example, physically take the laptop that contains the private key, and BOOM!
That's the jist. Private keys can be setup with passwords, but the person in control of the private key can change their key's pass/phrase at anytime after, so straight-forward key escrow strategies don't work. Inspecting the public key does not indicate the associated private key has any protection, and that's good insofar as one key not leaking information about its' counterpart.
That's where security ends, when 'it just has to work' because 'they're the ones paying/in charge'.
It is called an “AuthorizedKeysCommand” in `/etc/ssh/sshd_config`.
https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...
The more apt comparison should be made to other authentication technologies like, in particular, kerberos.
However, there's one big issue nobody is talking about: There is zero support for certificate authentication in any SSH clients for iOS and most people who have network access here also have iOS devices.
And even if there were support, just supporting the certificates alone is not enough - there would need to be some automatable way of getting a new certificate into an app as the whole idea of the certificates is that they are very short-lived (days or even hours if possible)
This is one of those things that drives me nuts as an experienced/old developer, seeing people type passwords for ssh/git/whatever several times per day. Sometimes there are tasks that require copying / checking some file on N servers, and these people seem to think that cannot be done in a shell script because the password needs to be entered interactively.
Then there's ssh port forwarding, X11 forwarding, etc... but its amazing how many people use ssh for years without so much as glancing at the man page.
The man page for the shell (man bash; man zsh) is a good place to start.
I can't explain with certainty why, but I think it's due to lack of discoverability, inconsistent conventions for flags and positionals, esoteric syntax for simple control flow, lack of errors/feedback for things like undefined variables, no scoping/namespaces, unclear type system (not asking for much, strings, bools and ints would suffice). That said, piping/streaming is amazing and often better than in modern languages.
In short, it's quite different from other imperative languages - the design feels arbitrary and the learnings non-transferable, even though I know it is useful and ubiquitous.
That's what autopw[1] is for. ;p
[1] https://github.com/jschauma/sshscan/blob/master/src/autopw
I wish they'd have been a standard that Microsoft, Apple, Google etc provided implementations of.
Really missed a great opportunity to add perhaps even more security than the server cert can(Because of how many users override it).
Come to think of it, website logins shouldn't exist either, that's way too common and I don't see why there's no PAKE based http basic auth feature.
I was able to recover it using the serial port but even after all this, the comfort of using SSH certificates on all of my nodes was enough to keep making me use it instead of Dropbear.
In the late 90s we came close to having this for the public internet as well but it never caught on. We paid the price with endless breaches and unmanageable credentials.
I guess that’s the small price to pay for speed of development without going through the international committee of ASN.1
Most everything do not enforce their DNS resolver to only return the DNSSEC-verified Answer RR.
Not that problem at all if you set the resolver to return only the DNSSEC-verified answer RRs; then again, most common websites would then stop working simply because they don’t use or have a proper setup of their DNSSEC overhead.
Most implementation of distribution of the SSH public keys are delivered under cover of TLS, IPSec, or variants of secured tunneling just because … because it IS A metadata.
ssh-keygen -f ca.key
# user key
ssh-keygen -s ca.key -I key_id /path/to/user_key.pub
# host key
ssh-keygen -s ca.key -I key_id -h /path/to/host_key.pub
Look into step-ca though, I've heard it's.. Okay? I don't know. It seems too complicated still - I'd rather stick with pubkey auth
What? This just sounds like you're doing it all wrong then blaming the tools.
Maybe I'm just thinking about it from my POV as a mainly hobbyist user of SSH, but I rotate my keys (not as frequently as I should, but I do), and I remove old ones from .authorized_keys files and I use different keypairs for different servers sometimes too. I never copy keys to other devices - I ssh-keygen on every device, and add my pub key to the authorized_keys file manually.
It isn't even "hard" to do it this way. Sure, it doesn't "scale" for big corps - but I don't need it to scale, so I'm not "doing SSH wrong" by not using certificates.
If you Google "ssh key rotation" you will find my article as a "featured snippet."
I don't think that I need a CA, as I only have personal and admin keys (two sets) and I'm flipping these once a quarter. Plus, I don't have expired entries in my authorized_keys (do CA users ever clean out authorized_keys?), and these are very clean as I see them regularly.
https://www.linuxjournal.com/content/ssh-key-rotation-posix-...
To me, setting up a CA is more effort than I would like.
I agree with you, for a regular user with a single client device (or two) its not worth it.
The biggest "you're doing it wrong" I see is people who disable host key verification because their servers' IPs change constantly. Do you want MITM?! Because this is how you get MITM! Might as well use Telnet for connections.
The audacity of mac users who think everyone uses a mac.
Linux desperately needs a standard for writing how tos. Even better would be definitive how tos for every common thing a user might need to do being posted and maintained on a distro owned site.
One of the things on my punch list is to get step CLI into upstream Fedora, Debian, and Ubuntu to make installation a bit more easy.
Having said that, `brew` appears once in passing in the entire article. This article is almost completely about ssh without regard to platform. There's nothing "audacious" about suggesting users use brew, so the anti Apple sniping here is entirely unwarranted.
There are orders of magnitudes of UX difference between how poor Windows is versus OSX and Linux, in my mind. OSX is still worse than Linux, in part because of the inconsistency of key mappings and lack of options to make things consistent.
I don't have encrypted drives on all my devices. I don't want to have to worry about what could happen if one of those gets lost/stolen. I'd rather not leave keys or certificates lying around.
Also, things sometimes go wrong and I need to get access to a server from a device I've never been on. It's nice to be able to do that. Passwords do that.
To be fair, I usually have a single VPS which I keep as locked down as possible that has VPN access to the server I really need. The VPS doesn't even need to be running most of the time. So I can spin it up to get access to the VPN, then ssh into the server with a password. If the VPS gets compromised, the VPN alone won't give an attacker immediate access to the server like it would if I left keys / certificates on there. I have to trust the VPS, and if it gets compromised without me noticing, and I then log in to my server, yeah, I'm SOL, but certificates don't solve that problem.
To what end? The threat model rekeying tries to protect against involves compromised authenticated client machines.
Once you have those, the attacker has shell on the server, and it's game over. There are these things called "advanced persistent threats" that have been in the news a lot already.
There's no need to open port 22 if tools like AWS Session Manager (and GCP's equivalent) are available to you.
Also, not all servers are on cloud platforms.
Not all servers are on cloud platforms, but there are somewhat comparable ways to shunt a serial or out-of-band management console over the network.
Not that I'd want to either, of course. Bringing an 1988 solution back to fix a conceptual 2022 problem does not sound like a great fix :)
I understand that in some workload types you want to have full autodeployment on servers, using ansible, kerberos, whatever. In that case interactive login is never needed.
But this is a very specific subset of 'servers' in my opinion. A lot of HN contributors work in this so this approach may work for them but it won't everywhere.