Show HN: Caddy-SSH | Hacker News Reader

Show HN: Caddy-SSH

caffeinatedwonders.com

280 pointsm_sahaf131 comments

Hi everyone! Author here

This has been my stress-reliever for the past ~2 years. I'm sticking around, so feel free to ask any questions.

Github Repo: https://github.com/mohammed90/caddy-ssh

Cool! (not a security professional) One of the reasons in my understanding that projects don't deviate from older languages was because they give you some control around compilation that would be necessary to thwart timing-based attacks. Does this consider timing-based attacks or is that at a lower-level library?

Excellent point; it does appear that the author did consider timing attacks in at least one location:

https://github.com/mohammed90/caddy-ssh/blob/master/internal...

To be fair, this bit is borrowed/forked from github.com/gliderlabs/ssh.

Curious why you would just copy the entire gilderlabs/ssh package into your repo instead of referencing it as a module? How do you plan to keep up to date with bug / security fixes?

Some of the changes necessary were too invasive/breaking to gliderlabs/ssh, such as https://github.com/gliderlabs/ssh/pull/161 so making a copy ended up making more sense, I think. That's my understanding from what Mohammed's told me, anyways.

That's right. Moreover, the activity no the repo has been a bit stagnant (no disrespect to the maintainers, they are likely busy with other projects or life). Other projects have opted to fork the repo, rewrite the module repo path, and maintain their fork (e.g. https://github.com/tailscale/ssh/, but I've seen many others too). I didn't want to maintain a fork, so a copy under internal/ in my own repo means it's firm-fork (between soft and hard) where I can confidently break its APIs I'm the only one depending on without worry. It will slowly morph to fit the needs of the project only.

The maintainers gliderlabs/ssh have plans for a new version with more ergonomic APIs but the work hasn't started yet. I didn't want to wait either.

Not very too level. I had to take that into consideration at some parts. For example, the static username_password provider calls a hasher defined in Caddy which uses `subtle.ConstantTimeCompare` function used. At other places, I don't return early (when possible) on auth failures to avoid timing attacks. That said, I'd love to know if there are places where I fell short.

I have reviewed some of the crypto code in the Go standard library that this is built with, and there is use of constant time primitives in there so it is at least possible and some attention has been spent on it.

Congratulations! Looks like a really cool project.

So, if I get this right, this should be a drop-in replacement for current SSH servers? I get that an adapter has to be built to support sshd config files, but is that the ultimate goal here? Is security the main selling point of this project?

Thank you! Indeed, you're right. The ultimate goal is to be drop-in replacement. There is a PR hanging waiting to be taken up. I might look at it once I finish furnishing the foundation, but others are welcome to take it up!

Hey! This is awesome. I’m the engineering manager for the Windows console subsystem team. I’d be happy to help out with your ConPTY issues!

Feel free to file a discussion issue over on GitHub at microsoft/terminal, or email me at duhowett@(corporate domain name).

I suspect what you need is CreatePseudoConsoleAsUser… which we should have offered as a public API.

Can you point the right person to this internally for the real OpenSSH shipped with Windows? I'm actually curious how licensing works out for 3rd party servers like Caddy-SSH.

Licensing / Multi-user access / CAL | https://github.com/PowerShell/Win32-OpenSSH/issues/926 (Oct 2017)

Am I understanding that the public keys used for authorization (authorized_keys) can come from a centralized source? Being able to add and remove authorized keys from say Redis or Consul centrally would be extremely useful from a management perspective.

Obviously, security of that Redis or Consul would have to be tight and prevent public access.

That's right! Currently such (static) keys are loaded at provision-time, when the server is first booting up, but there's nothing preventing it from being lazy-loaded at authentication-time. Of course loading them at authentication-time incurs latency as tax. Nothing prevents two separate modules from existing: one eager-loads the keys, and another lazy-loads them.

Slightly off-topic, but OpenSSH already supports this via `AuthorizedKeysCommand`: https://man.openbsd.org/sshd_config#AuthorizedKeysCommand

Oh really interesting. Any good guides you know of how to implement AuthorizedKeysCommand with Redis for example?

I'm not really an expert on redis. Perhaps one can fashion something with `redis-cli`? Or write a program that links the redis client library.

Hi @nodesocket, you can take a look how we solved it with Theo https://theoapp.readthedocs.io/en/latest/index.html It supports fine hosts/users grants - i.e. I can connect as "dev" user on servers "node1" and "node2" but not on "node3" - and it leverages asymmetric key signing to validate the public SSH keys. Theo supports mysql/mariadb/sqlite/postgresql(experimental) for storing data and redis/memecached for caching. Happy to answer any further questions!

Hmm interesting.

Considering memory corruption vulnerabilities have been pretty much the vast rarity in the OpenSSH codebase, and other classes of vulnerabilties have been more common, I'm curious to know what you're doing to address those classes of vulnerabilities as well as support for other best practices like SSH certificates? [2]

I think I see the appeal of an entirely memory-safe OS running an entirely memory-safe SSH implementation because if we're talking about eliminating that class of vulnerability, you might as well go the whole hog - I just don't see that close to the point of being battle tested yet nor taking care of stuff like side channel attacks or tricking people into thinking their TOFU isn't smelly. I like your project because I think that this stuff is important, I think it's just like "how do you improve on the original" may actually be more than just eliminating one class of vuln.

[1] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=openssh [2] https://smallstep.com/blog/use-ssh-certificates/

What's the benefit of this being built on top of Caddy, which unless I'm mistaken has historically been an HTTP(S) server? Maybe there's someone intrinsically great about how Caddy works for this, but on first glance it feels like the scope creep in servers like Apache, or Curl's support for querying every data source under the sun. It can be handy having a single tool that does everything, but at the same time that gives an ever larger attack surface where suddenly your machine has been compromised via instructions from an MQTT server because Curl was installed.

Caddy v2 is actually designed as an "app platform". The HTTP app is just one such type of app, and is what Caddy ships with by default. But it can be extended to do _anything_.

See https://caddyserver.com/docs/architecture

You don't have to run both the HTTP app and the SSH app in the same process, you could have two separate Caddy processes configured to each do their own thing, if you're at all concerned.

Isn't this like a super-server in the spirit of inetd?

I guess, except Caddy doesn't spawn any processes, it just "starts apps" which are configured in-process, pure Go.

Another example Caddy app is https://github.com/mholt/caddy-l4 which lets you do arbitrary TCP/UDP handling/proxying.

There's a list of available apps here https://caddyserver.com/docs/json/apps/ (the SSH app is not there yet, should be soon)

I think it's fine for a conceptual understanding. The Go runtime looks a lot like an OS, mapping G (goroutines) to Ms (worker threads), removing goroutines from the workers when they block in syscalls, waking up goroutines when a lock they want is available or a syscall finishes, etc.

In the end it really depends on what you think of inetd as. If you think of it as "I install this one server and now I have finger and gopher!" then that seems similar to Caddy's applications. If you think of it as "whenever a TCP connection arrives, the fork() syscall is used to create a process to handle that session, file descriptors 0 and 1 are connected to the TCP socket, and then exec() according to the global config is run to handle the protocol", then ... no it's not the same as inetd.

> The ISRG estimates ~80% of the vulnerabilities exploited in the wild are memory safety bugs.

Okay, but 1. How many vulnerabilities has openssh shipped, and 2. How many of those were memory issues? I would usually be tentatively on board, but you're competing with the OpenBSD folks, who have a shockingly good track record regardless of using C. No offense, but you could write in a formally verified Ada subset and I'd still hesitate to replace my SSH daemon.

(FWIW, I say all of this hoping to be wrong; an alternative implementation, if equally secure, would be great to have.)

Not sure about the statistics but one of the most memorable vulnerabilities in recent history, Heartbleed, was exactly this, a OpenSSH memory safety issue.

No. Heartbleed was OpenSSL, not OpenSSH.

Heartbleed was OpenSSL, not OpenSSH.

That was in OpenSSL not OpenSSH, two entirely different beasts.

Sibling commenters have already pointed out you're confusing OpenSSH with OpenSSL.

For additional context, since the gp mentioned that:

> the OpenBSD folks, who have a shockingly good track record

Not only is OpenSSH a BSD project, but so is LibreSSL (the OpenSSL fork that was a response to Heartbleed). Before LibreSSL, BSD folk were also working on assl - similarly an OpenSSL alternative motivated by the BSD guys having concerns about OpenSSL's codebase. So they very much have a solid track record of being pro-actively on top of this type of stuff.

However, on that note it's worth taking a moment to look at recent OpenSSL bugs and compare LibreSSL and see that unsurprisingly (to me anyway) the OpenBSD people did not magically fix the bugs in hairy C code. Once Libressl had taken out the unnecessary stamp collecting from OpenSSL (crypto algorithms you will never need, feature switches left over from experiments a decade ago, etc.) it did not in fact evolve the rest of the codebase as much as you might imagine.

If you've found a bug in pre-fork OpenSSL core material (not the stamp collecting), chances are that it still works in all the forks, none of them have the beyond wizard level competence to rewrite somebody's micro-optimised ECDHE implementation in a way that non-wizards can review, so even if they replaced it they'd just be swapping possibly-incorrect wizard magic for different possibly-incorrect wizard magic and what's the point of that?

i think the point is that not everybody can be quite as awesome as the OpenBSD folks, and memory-safe languages can allow mortals to write secure software.

if this helps us get away from the scenario where literally everybody uses the same SSH server implementation, i think it's a great thing.

Another aspect is that security software has additional requirements that most other software doesn't, and that can be a problem. My impression is that the crypto community generally regards the Go crypto stack as fairly algorithmically solid, but it hasn't necessarily been hammered on to the n'th degree for things like timing attacks. I'm comfortable using it in my usecases, but I'd be a bit nervous about deploying it to my high-security stuff. Things like reaching out to the network for crypto keys is also a convenient feature in some ways but something I'd want to see much, much hardened compared to what I might ask of a non-security program doing something similar.

That said, I don't say this to discourage the project. I wish it all the best and hope that they continue to work out the issues involved. I'm speaking more here to dampen the set of programmers and/or ops I know who will read something like this and immediately begin converting their entire fleet, because this looks new and hot and uses $TECH I like and must therefore be better than the old and busted using $BADTECH I don't like. Hey, I'm on the cutting edge of hating C and thinking almost anything written in it should be banned from the network, but OpenSSH does have a pretty good track record and that shouldn't be chucked overboard for new untested hotness.

Best wishes to Caddy-SSH. I would encourage a bit of trying it out, just, you know, don't go crazy. I don't think they'd want you to either!

Good point. Software monoculture is bad though (OpenSSL) and choice is good.

That said, I wonder how many crypto libraries are shared between Caddy-SSH and OpenSSH?

Finally, I really hope this is a separate daemon. SSH served by a web server invokes a certain systemd feeling.

> I wonder how many crypto libraries are shared between Caddy-SSH and OpenSSH

None, because Caddy-SSH is written in Go, and uses Go stdlib crypto libs which are their own thing. See https://pkg.go.dev/golang.org/x/crypto

> Finally, I really hope this is a separate daemon. SSH served by a web server invokes a certain systemd feeling

That's entirely up to you. Caddy v2 is designed as an "app platform"; the HTTP app is just one such app that Caddy happens to ship with, but anything else can be plugged in, like an SSH app in this case. You don't have to run both HTTP and SSH in the same process, you could run two instances of Caddy each with their own purpose, if you feel like. https://caddyserver.com/docs/architecture

Not sure what you mean with systemd. All big parts of it run as separate, locked down processes.

But yeah, just for the time when you're hammered with traffic that you want to investigate, you don't want your web server and SSH session to go through the same loop.

* CVE-2019-16905: https://www.cvedetails.com/cve/CVE-2019-16905/

* CVE-2016-10012: https://www.cvedetails.com/cve/CVE-2016-10012/

* CVE-2016-0777: https://www.cvedetails.com/cve/CVE-2016-0777/

A list of all CVE is available here: https://www.cvedetails.com/vulnerability-list/vendor_id-97/p...

The 2019 bug refers to a bug in an explicitly experimental area of SSH code (post-quantum crypto XMSS)

most of the other server-side bugs on the list relate to non-standard configs (of varying obscurity)

the fact that - in an era of increasingly hostile cyber threats - years had gone by without any serious threats says it all really.

for most people, they should just run normal SSH, enable public-key only authentication and get on with their lives.

I see no reason why anyone should jump ship to some shiny new untested ssh server. Especially as , when others have pointed out, the people behind OpenSSH are the same people behind OpenBSd and LibreSSL.

There are obviously more than just those tho.

Not that I really disagree with the conclusion. There's not a ton of pre-auth attack surface.

For reference, the first is with an experimental feature, second is a local privilege escalation, and the third allowed a malicious server to read the clients keys. For comparison, here's some CVEs from software written in Go:

Caddy CVE: https://www.cvedetails.com/cve/CVE-2018-21246/ Kubernetes CVEs: https://www.cvedetails.com/vulnerability-list/vendor_id-1586...

No silver bullets! Though some have less lead in them. OpenSSH, along with other software developed by OpenBSD, has earned the highest level of trust in something written in C. C can be dangerous, but there is a chasm between such careful works and the typical offenders like OpenSSL (OpenBSD's alternative is LibreSSL, confusingly).

This could also be a replacement for things other than openssh, like dropbear. Though golang binary size might be a barrier for that.

I'm assuming this being Go and the mention of being extensible through Caddy modules (in-process API) that this is a monolithic, non-privilege-separated server, which likely makes it quite a bit worse than OpenSSH sshd in terms of defense in depth.

Ceteris paribus, I would feel safer with OpenSSH than with a new SSH server probably for several years after the release of that new server, simply because OpenSSH is so scrutinized and because there's more things that go wrong with SSH than memory safety bugs. It's been well over a decade since the last exploitable OpenSSH bug. But it would also be a very good thing to have a memory-safe trusted SSH server. This isn't an easy call.

I think it will just take some time. I'm glad we have a solid start on a new alternative, and hopefully people will use it, scrutinize it, and improve it. It's a little unsettling to realize that when the next vuln in OpenSSL is discovered, it will likely effect 99% of servers on the Internet. (Or something like that.) I'd feel a little better if that percentage is brought down by something that isn't susceptible to those kinds of vulns in the first place.

(D'oh, I meant OpenSSH, but it's too late to edit.)

Note that outsourcing your key list to a live GitHub URL gives Microsoft unfettered access to your box, should they (or anyone who can compel them, such as the US armed forces) ever want or need it.

If you wouldn't use Microsoft SSO for local login, you should not thus configure your sshd that way.

How does outsourcing your key list give ms access to my box? Presumably you mean because they also have the private key?

Because they can serve any list of public keys they want; they can return a blank list and lock you out, or append another key at will.

No, the keys on GitHub are only the public portion.

If your sshd consults Microsoft servers to check a key, Microsoft can serve it any file they wish (or are forced to), such as a list of Equation Group pubkeys.

This looks very interesting. Is there any support (or plans) for SSH certificates? They help to manage some of the revocation and access control challenges, as well as the issues around trust-on-first-use and similar. (And also the fragility of syncing around authorized-keys files, or relying on LDAP for login, in infrastructure type environments).

I had to think of exactly this post as well and I started to wonder, whether Caddy-SSH couldn't even handle this using Caddy's built-in ACME support?

It could, if there was an ACME server doling out SSH certificates.

And Caddy can get certificates via more than just ACME, it's completely extensible, so it will work with... anything I guess.

Although the word "Certificate" is in the name of this feature and the expansion of ACME, they don't really have much in common.

ACME is similar to, and different from the Internet's other certificate protocols, both in ways that are not helpful for a SSH server.

Firstly ACME is similar in that it's about X.509 certificates roughly PKIX flavoured, whereas certificates in SSH don't use X.509 and thus don't use PKIX

Then it's different in that ACME solves the proof-of-control problem for the Web PKI, it was actually built in parallel with what became the "10 Blessed Methods" for the Web PKI (but today there are not ten of them). But almost nobody wants public SSH servers for random people from the Internet to access, so the Web PKI is largely irrelevant to certificates for SSH servers.

[ The Web PKI is the name for the PKI that makes your web browser work, unlike SSH you actually do use web browsers to connect to random public stuff - lots of other things rely on the Web PKI, not just the Web but there are good reasons it is called the Web PKI anyway ]

The plans are there! I focused on implementing the absolute necessary parts of every layer before taking round-2 for the more in-depth implementation. I actually have both the linked thread and the article saved aside to study when I'm ready to implement certificate-based authentication. Knowing Gitea already has it implemented, I had plans to study their implementation to know what I'm venturing into. If anybody else is interested in picking it up, please feel free to tackle it! I'd love to see that PR.

I've created tracking issue: https://github.com/mohammed90/caddy-ssh/issues/10

Great to hear, and nice work with the project so far!

I think it definitely makes sense to focus on getting the minimum viable feature set ready first - the only reason for suggesting certificates was it seemed like the kind of area where a caddy-based setup could really help simplify the process for users, and make it easier. Users hate trying to manually define quirky configs, but I could see some well-thought-through caddyfile syntax based on extractors and pattern matching with some useful defaults proving very useful for a range of scenarios that would make it very quick to deploy.

Things like "fetch the username from this field, let the user log into anything with that username", then expanding to say "... As long as the server thinks it's in a group that the user certificate lists", and "let them log in as any user, but only to hosts in a group listed here in the certificate". With user, group and hostname you likely have a lot of what people would need, aside from sane defaults like checking expiry and validity, checking signature on the certificate, and some kind of usable revocation checking, which is always an annoying pain point for any long lived certificate-based system, but which a system like caddy could make into child's play!

You're not wrong -- Caddy has a PKI app built-in, which is basically a wrapper around Smallstep's CA libs. That could be used to issue SSH certs as well, probably.

Pluggable auth providers for SSH sounds awesome, though I can't think of a particular use for it right now!

Ironically, these already exist for OpenSSH and have been developed for Linux and UNIX for decades. It's funny how people keep reinventing things. You can use them in your /etc/ssh/sshd_config with "UsePam yes", but, to your point!, generally the built-in auth is a better option for a smaller surface area, unless you specifically know you need PAM.

They are literally called Pluggable Authentication Modules (PAM) and it's a core part of any Linux distribution. (I'm not aware of any modern UNIX or Linux that do not offer PAM, but perhaps some embedded distros.)

They can be configured for many types of server software, not just OpenSSH. (For example, mail servers like Dovecot and Postfix support PAM, database servers like Postgresql and MySQL, etc.)

https://tldp.org/HOWTO/User-Authentication-HOWTO/x115.html

> It's funny how people keep reinventing things.

That's not exactly fair. The entire point of this exercise is to move away from C code, by implementing it in a memory safe language (Go).

Since PAM uses shared-libraries to operate, that's fundamentally incompatible here since Go is statically compiled (unless you use some CGO like in https://github.com/msteinert/pam) so implementing auth via Caddy's module system is the way to go for this project.

Memory safety won't save you from logic bugs, which is extremely prevalent in authentication exploits.

No arguments there, but it's still much safer to start with a memory-safe language as a base.

While it's true, most languages we move to today as memory-safe system languages do help with logic bugs. Full struct initialising, actually typed and checked enums, pattern matching, ADTs, etc. make logic bugs less likely.

> That's not exactly fair. The entire point of this exercise is to move away from C code, by implementing it in a memory safe language (Go).

Ok, I'll go along with that, but being memory safe is no panacea either.

> Go is statically compiled

As soon as you use os or net, you're almost certainly dynamically compiled unless you take deliberate steps not to be. (You can check your final binary with ldd to see if it's dynamic or not.)

Here's a few things you should do to be sure you're compiling statically:

    CGO_ENABLED=0 go build -tags osusergo,netgo

Also, you can't use the race detector or explicitly C libraries like SQLite. And, be careful -- if you use CGO and try to compile statically, then you might end up including glibc.

I'm aware of PAM, though I always assumed there was a relatively high barrier to entry, and C would be required?

What is involved in using PAM on Windows and Mac?

Nothing on a mac since MacOS already uses PAM

How do you do GSSAPI with PAM? (Does OpenBSD support PAM now?)

OpenBSD does not use PAM. Closest would be BSDauth.

PAM has limitations that start getting in the way of more interesting setups. It's also a very specific, integrated environment that's not trivial to extend beyond standard credential checks. I've written PAM modules and I'm happy we're reinventing it in different ways.

I feel like most practical situations which require this end up just routing through the OS user management and using PAM, but providing git-ssh access is an obvious case where that's not desirable or even necessarily possible.

I'm a bit curious what this ends up meaning for projects like GitLab and Gitea. Gitea's docs [1] indicate that it can either use the system sshd or one that it "provides", but it's not clear whether the provided one is Go-implemented or just a statically-compiled instance that it runs as a subprocess.

[1]: https://docs.gitea.io/en-us/faq/#ssh-common-errors

I'm not sure I fully understand, what's the advantage of getting Caddy involved here versus a fully standalone app?

Given the name I'd at first figured it was an official Caddy project, but that does not seem to be the case.

Names are hard. I did consider changing the name but couldn't find another one. Feel free to pitch names on GitHub.

Caddy provides the module system and the config management backed by powerful REST API for config query and patching. If it wasn't for Caddy, I would've had to build all of that from scratch. That's too much to own.

I filed an issue to suggest caddysshack.

Speaking of sane defaults, I wonder how many people are aware that Caddy serves dotfiles by default.

That's incorrect. Caddy does not serve any files by default. You have to explicitly enable file serving in your config.

Sorry Matt, didn't want to be rude. But yes, it does. Using the file_serve directive, dotfiles are served by default.

Here are some random domains taken from the Caddy Forum threads:

https://lastfree.space/.git/HEAD

https://www.aspecthq.com/.git/HEAD

Those people are unaware of serving their git (or .env files or other secrets) over the internet, probably because are used to other webservers that just block this behaviour by default.

And it's ok to not do or be like the others, but this should at least be mentioned in the docs.

What you initially stated implies a completely different meaning.

You initially made the implication that Caddy would enable the file server implicitly when it does not. That was what I interpreted upon first reading.

Your correction states that dotfiles are served using the file_server directive which is correct and and example is given in hiding the .git folder [1]

[1] https://caddyserver.com/docs/caddyfile/directives/file_serve...

With all due respect, you're wrong. If Caddy is serving any files, the user explicitly enabled that functionality. And when enabled, it only serves files within the root directory specified by the user, or the current working directory, and the docs are very clear about this: https://caddyserver.com/docs/modules/http.handlers.file_serv...

I stand corrected about the docs, somehow I missed the additional clarifications.

Biggest obstacle to proper SSH usage tends to be the creation and management of people's private keys. The number of times people have to google for instructions for doing this right is insane. Let's get someone working on that.

true story

Teleport I feel solves this well, from an organisation perspective

Cool - was not aware of that service. Will have to check it out.

At my org, if I use teleport, I have to MFA in via a UI, and they offer zero other alternatives.

The teleport daemon falls over due to memory stress and other issues that sshd doesn't fall over during, and when do you often need CLI access to a node? When it's under stress....

Goodbye automation.

It's a good idea, but it's made ssh and the problems I solve with ssh worse for me.

First and foremost, congratulations on bringing the project to this stage - I think it's an impressive piece of work.

I am in no way qualified to trample on your parade but two things came to my mind that pinch a personal nerve of mine and I would really like to have alleviated by you or the folks who know that stuff:

- if your Goal was "secure by default", why did you allow passwords in the first place? Following Caddys recipe would be more like SSH-Keys only, wouldn't it? Is there a reason other than compatibility?

- In that same avenue? Why allow such a thing as downloading authorized keys from a third party? Domain takeovers or account compromises on say Github are a thing - so again while it may be a nice usability aspect isn't that contrary to the secure by default pradigm?

Again thank you for your work and congratulations on the project - those above are just honest questions that came to mind which I would really like to be educated on

Passwords are not insecure. Certs have tradeoffs.

I don't read it as "passwords are insecure" but as having defaults that force people into using it in ways that are harder to mess up.

Re-using an insecure password is easy, using a key wrong is harder.

re-using an insecure key is just as easy https://github.com/mikalv/anything2ed25519

I understand what you are saying but in my experience the opposite can be equally true. Unless every system in your org is managing the authorized_keys for every account with automation and hourly validation one can end up with rogue keys, forgotten keys, unmanaged keys. SSH has no concept of identity in this regard. If I temporarily have sudo on a system I can append any random public key into the authorized_keys of any account. It gets even worse if that system allows passwordless sudo. Now there is an audit trail that showed you made a reckless change when it was really me. This risk gets exponentially worse if the system I append a random key into is a command and control or configuration management system. One doesn't even really need sudo for this to be a problem given how much power non root accounts have over applications, deployments, builds and secret management.

Is authorized keys important here? If I send the server my password, it has my password. If I send a server my public key, that's a lot less severe. If an attacker is on the server I expect that the difference is obvious?

Both are vulnerable to phishing via malicious scripts. One could even argue that keys are more dangerous if the attacker can append their own key into authorized_keys one host at a time. Some might be surprised how incredibly easy it is to get a percentage of technical people on an email distribution to execute a script without fully understanding it and this is even leaving things out like compromising npm repositories. Said script can drop a key into authorized_keys or drop a custom key and create an outbound session invoking a gateway port allowing the attack to ssh to the host even if they have no inbound ports open. From there one can repeat the process as access is gained first at the workstation or laptop, then the jump host then build servers, repository servers, staging servers and ultimately production servers.

SSH is a power communications tool. It is equally powerful to both the authorized user and to attackers in my experience. It is often poorly managed if even managed at all even in the most sensitive environments in some of the most high profile organizations.

How would you phish my private key?

I would not. I would phish you running a script that adds my public key to your authorized_keys and would ssh from your client in the background creating a gateway port back to you. At that point I can get your ssh keys and just about anything else. If systems in your org allow ssh multiplexing default is enabled then I can also piggy-back your sessions bypassing MFA/2FA.

I say you but I mean most people. I'm certain that you specifically would not fall for it but about 10% [1] of technical people will.

[1] - stats based on past career and testing of a large technical population.

I'm not sure I understand the attack. I connect to a server that you, the attacker, are on. How does your attack get my private key? Assuming I'm not forwarding my key, and you're not on my laptop.

You run a script that has obfuscated code. The script is "broken" and has an easy problem for you to identify for me. You feel good about finding my amateur mistake.

In the background the script has dropped my public ssh key either into your authorized_keys if you have sshd running, or into a random location if you do not and launches sshd as you in the background if it is not running. It can be on a high random port listening on 127.0.0.1, this is fine.

The script also backgrounds an outbound connection to a machine I control enabling a gateway port that I can use to SSH back to your machine even if you are behind a firewall. It will tell me if you have ssh listening on 22 or on the loopback and what port. My tunneled ssh connection will utilize that ssh key I appended or dropped. I am not on your machine as you. That back-grounded connection can also be added to your .[a-z]* shell login files based on the shell you are using at the time so that if you reboot the tunnel is recreated.

Now I am on your machine as you and I can grab your private keys, ride along your existing SSH channels that you have authenticated with your 2FA/MFA provider assuming the hosts you connect to leave multiplexing enabled, connect to your browser and watch the console http session data and so much more.

This scenario will technically work on most peoples setup. About 10% of people will run the script. No hacker tools or malware required, just an obfuscated script. This will never be detected by anti-malware software. The script language will depend on what I think your organization uses. It could be python, perl, ruby, go, java, bash, anything really.

The only technical mitigating control would be to limit where the person can make an outbound connection. i.e. pre-approve TCP destinations. If a laptop or workstation can SSH to a random cloud provider then the script will work.

You're not on my machine. You're on the server. You still don't have my private key.

What I described above puts me on your machine. This is not theory, it has been tested against many people. Legally of course.

So you were vague about where I execute the script. If I execute the script on my client, obviously you have my key, but that's not interesting. If I execute the script on my server, you do not have my private key, period.

This whole "I can ssh back to your machine past your firewall" is not a thing I understand, I am not aware of that capability in SSH.

I've received the private component attached via email before. People find a way to mess up.

Passwords are a shared secret. I'm not sure I'd choose the word "insecure" but only because it's too black and white when the reality is more nuanced. They are clearly worse than doing public key auth which is the alternative and is Mandatory To Implement in the SSH (SecSH) RFCs.

Nobody said anything about requiring certificates in SSH, just public keys as authentication.

Strong aPAKES can allow passwords to not be shared secrets.

Of course they're super niche now and none of them are standardized so they're basically impossible to use, and SSH doesn't support them, so this is a pedantic nitpick instead of some sort of insightful observation.

No they don't. Passwords are worse than certificates or keypairs, categorically.

> No they don't

SSH into one of your boxes from my machine real quick.

What's the use case for using your personal unix account on someone else's computer?

I've done it from work computers where the USB ports are disabled but I want to hit my personal dev machine at home to get at my notes and calendar.

I've established impromptu SSH tunnels from other people's machines to my local network so that I could watch my media on their TV.

If you dropped me naked on the other side of the planet I could get a copy of my identity documentation and access my email, bank accounts, etc from any internet connected machine I find.

You get that passwords over first-time SSH from untrusted computers or untrusted networks aren't safe at all, right? That posting those passwords is literally a sport at hacker conferences, and has been for over 2 decades?

This whole thread is a little alarming.

You get that waking outside isn't safe at all right? Mugging is as old as civilization and don't get me started on cars.

Life is about trade offs, if someone really wants to spend the time to get access to my home dev box then I may have to spend a couple days on the phone with the bank and restoring from my offline backups. Big whoop.

Your home is likely insecure from my standards. Do you have a firearm at the ready? Do you know how to use it? Does your family have codewords to communicate without letting others on to your plan? How hard is it to kick in your doors? Not just the front door, but the bedrooms. Do you have a dog to wake you in the night? How stocked are you, can you last a month with no resupply? Do you even have a panic room?

I protect what matters beacuse you can't protect everything.

How are we still talking about this? SSH doesn't work the way you appear to think it does. Passwords don't solve the first-use problem; in fact, the first-use problem makes passwords much less safe than keys.

Yes and your cheap hollow core door doesn't solve the ending the life of everyone you care about problem. Why are you more concerned about passwords than that?

I know the risks well, I don't find them to be worth the hassle of avoiding them.

Iirc ssh passwords aren't sent in the clear. Is that really the case?

Edit: I looked it up to confirm. First time password is encrypted analogous to tls

No, it's not analogous to TLS. TLS has trust anchors: the key exchange in a TLS handshake is secure, even on first contact with a server. The key exchange in a first-contact SSH handshake is trivially MITM'd. An attacker with control of your network (or, obviously, your machine) can simply steal your password.

I'm sorry I should have said it uses diffie hellman key exchange. And you are correct it can be mitm. I see that as its primary intrinsic weakness. (Password entropy, storage, are separate problems)

I would contrast this with the weakness of keys being that if the devs keys are compromised so are all the other servers he has access too. (I can memorize my passwords, or write them on a note card. Say what you will about that, it's out of band.)

In light of that the question as to what's best is your threat model. Poor opsec per dev or an upstream network sniffer.

Thanks for all your input all over this thread. I'm revisiting my convictions.

Would you disagree with I've said?

But you've now swapped out part of the authentication. If you want to claim that, for some threat models, "One password per server" is better than "One key" then, sure, so use "One key per server" and now keys are better again.

Also - I suspect this Caddy server doesn't support it, but OpenSSH does - you can use FIDO and then the keys physically are objects in the real world, from say Yubico or a dozen other vendors so now "losing the keys" is like losing your office keys, except that when they give you a new one they can trivially make the old one stop working.

A users keys are unlikely to be compromised singly. You would get them all. Whatever malware exfils the keys would take all that it finds.

My mail server crashed while I was on vacation without my laptop.

Of all the use cases I can imagine, enabling password logins just so I can meaningfully type my password into strange computers is among the least compelling.

How do you log into a server for the first time without a password? How do you log into a server from a brand new machine without access to your previous machine or sharing certs?

Keep following your own logic. If you don't have a preexisting key relationship and you can't trust the network, are passwords safe? [Y/N]

> and you can't trust the network

Why is this a requirement? Not having a configured machine does not imply an untrusted network. Maybe all my computers recently and unexpectedly got wiped and I'm trying to rebuild my home network.

Repeat exercise, but with untrusted computer. [Y/N]

If you trust literally every component involved in the exchange, is rsh safe? [Y/N]

If you cannot trust the network, where is the advantage of keypairs over passwords?

If you have a one-time password, worse case is that some man in the middle gets that password.

If you engage in a proof of private key ownership for your login, a man in the middle can use that exchange to log into another server that has the same public key.

He can do it once, maybe, if you don't authenticate the server. With a password, he can do it any number of time, to any number of targets, and there's no way to catch it.

Yes, it is still not perfect, but here's the advantage.

First, if you use an SSH CA, you don't necessarily have the first-use problem at all; that's part of the point of SSH CAs. You've resolved that problem the same way TLS does.

Second, an attacker targeting your keypair-backed SSH session on an insecure first-use gets your session; against a password, they get your password, which is strictly worse.

It's not my claim that keypairs neatly solve the first-use problem with SSH (though: that problem can be solved, with more keypairs). It's that keys are categorically better than passwords. Which, of course, they are.

The alarming thing about this thread is that there's a couple people here that clearly seem to believe logging in with a password to a "new" SSH server is safe. It's literally the basis for the "Wall of Sheep" at hacker conferences; they were doing it at Usenix when I was there in 1998.

> If you engage in a proof of private key ownership for your login, a man in the middle can use that exchange to log into another server that has the same public key.

No, this is not true. The proof of private key ownership is bound to that specific SSH session (identified by some shared secret established via DH). This means the attackers options are: MITM the DH: Then your authentication won't work against any server. Don't MITM the DH but forward your authentication to the wrong server: Then you're on the wrong server (you might not notice), but the attacker cannot look into your session or modify anything.

Depends on how the server was setup. Ubuntu has offered importing public keys from github for quite a while in its installer. If you're using terraform/ansible/some other IaC tool, adding a key is probably just 1-2 lines of code. Some OS, like Guix or NixOS, also allow you to define them in a config file. Worst case you can just do something like `curl -L github.com/username.keys >> .ssh/authorized_keys`.

This is also part of the reason big fleet operators like SSH CAs, since they work for host keys and solve the key continuity problem.

When you install Ubuntu it asks you for your Github username and pulls your public key from there, making having the corresponding private key the only way to log into the system.

If you lose all your key material, then I wouldn't expect for you to ever be able to log into the system. Passwords have the same problem; if there's some machine you can only remotely connect to and you forget your password, you simply won't be accessing that computer anymore.

So there are three different scenarios wrapped up in this question and they're all interesting.

A. The server is new, and I'm setting it up presumably remotely e.g. in "the cloud". Ideally the autonomous setup should provision keys for me in this case, since they're public it's fine to even publish scripts which do this. However it's true that there are a lot of systems out there which instead give you a password for first login (or worse they have a fixed default!) which is sad.

B. The server has existing users, but not me. If we're not using certificates then I need an existing user to authorise me to use the machine, providing public keys is no different than anything else they might fill out, like my name, they will need a trustworthy source for all of it. For systems I have no other prior relationship to beyond that I am, as you see, tialaramex, I tell them to add the github keys for tialaramex, which are of course published. I will replace those after successfully connecting to their SSH server.

C. I've used this server but not from this machine. FIDO resident keys solve this (you can extract the resident key to a new client from your physical device, the device still needs to be present to actually authenticate) but I will also use SSH to connect to my home bastion and have that authenticate while I do setup once. I can SSH from my phone, so if this can't work then somehow I don't have my phone or access to my home, setting up SSH keys is a low priority in that case.

However it's true that none of this is yet ordinary.

Good passwords are usually not insecure. But if you give your password to a host that you don't 100% trust (i.e. that you didn't setup yourself locally) you have potentially shared that password and if you've reused it anywhere the other wheres are potentially compromised.

If you give you public key to be installed to allow access, even if the host (or any system you are using to build/interact with it) is compromised your private key and any other hosts that accept authentication that way are fine.

How much difference this makes to you depends upon your threat model and how much extra threat you are willing to accept for a little convenience, of course, but key based auth is demonstrably more secure than passwords for some circumstances and no less secure in others.

Then again given that very few people bother actually checking host fingerprints on first connection, then proceed to send important data to that unverified host, is the password/key issue the first thing we need to fix?

> - if your Goal was "secure by default", why did you allow passwords in the first place? Following Caddys recipe would be more like SSH-Keys only, wouldn't it? Is there a reason other than compatibility?

Compatibility was the consideration. The presence of the password based authentication does not mean it is enabled by default. In fact, if you don't enable any of the authentication flows and don't explicitly set `no_client_auth` to `true`, the server will not accept any request at all! You have to explicitly set `no_client_auth` to `true` for the server to allow unauthenticated users to login. The default is: none of the authentication methods are configured, and `no_client_auth` is false (meaning do not allow clients without authentication).

> - In that same avenue? Why allow such a thing as downloading authorized keys from a third party? Domain takeovers or account compromises on say Github are a thing - so again while it may be a nice usability aspect isn't that contrary to the secure by default pradigm?

That was just an example, but the implementation doesn't only support github. It supports any HTTP/S URL you provide. The point was that the source of the keys may even be an internal HTTPS service within your network that applies its own logic for keys issuance and generation.

Reply on news.ycombinator.com