OpenSSH versus SSH | Hacker News Reader

OpenSSH versus SSH

utcc.utoronto.ca

183 pointsingve96 comments

This protocol/standard vs implementation is not uncommon. BIND was synonymous with DNS for a long time, to the point that I'd argue that some of the things specified in the DNS RFCs should have been considered BIND-specific implementation details. For example, RFC 1035 specifies the syntax of zone files used by BIND. I'd argue that even AXFR (zone transfers) as a replication mechanism should have been a BIND implementation detail.

This was Daniel Bernstein's argument, back when he was doing djbdns, and I believe it got him kicked off the IETF DNS list (he was, in this one instance, correct; I'm still confused about why zone files are a part of the DNS standard).

djb is a great lesson in how "being an asshole" can be counterproductive.

> counterproductive

If the goal of a committee member is merely to be included in the committee. Otherwise, if the committee is merely operating as a rubber stamp for every one of it's members proposals, then there may not be much productive value in remaining there. Apparently that feeling was mutual.

There is a middle ground to he found, in upholding high standards without getting kicked off the team

True. I think perhaps the model of IETF design-by-committee, with giant industry players who wish to sink specifications under the weight of risky and unnecessary features, is much like the "too many cooks" problem. It's better to have smaller groups of renowned experts create minimal protocols rather than force another ASN.1 or IPsec on the world.

I don't really think that stereotype holds, and, especially in the DNS case, it often seemed like random university IT administrators were the people monkeywrenching the specifications.

ASN.1 is tiny. It's the encoding rules which add complications.

The rules for a turing machine are tiny... it's potential output is not. When you're looking for a small subset of well defined behaviors in order to maintain compatibility and application security those complications matter a lot.

The IETF does not force ASN.1 on anyone. There are many Internet protocols (i.e., with RFCs published by the IETF) that use things other than ASN.1, such as, for example:

  - lots of protocols like HTTP and SMTP that use ABNF and text,
  - protocols like DNS, SSHv2, and TLS that use ad-hoc syntax and encoding,
  - NFSv4 (which uses ONC RPC),
  - protocols like XMPP that use XML

and many others.

The IETF doesn't care very much about those details because by the time a protocol has been brought to them for standardization it's already too late to insist on radical changes unless those radical changes are necessary to make the protocol secure or otherwise viable on the Internet.

In fact, a pretty significant driver of bad IETF security standardization is hatred of ASN.1. Dislike of X.509 is like 95%+ of the motivation for DNSSEC.

As an maintainer/implementor of an ASN.1 compiler and an x.509 stack, I have to say I rather like both, ASN.1 and x.509. Well, except for the x.500 naming bit of x.509, which like x.400 naming should never have happened.

Hatred of ASN.1 is misplaced and has been extremely counter-productive. Hatred of BER/DER/CER and all TLV encoding schemes ever is fair (so let's adopt OER!). Hatred of ASN.1 has led to things like Protocol Buffers, which... uses a TLV encoding because... probably the designers didn't really take the time to understand what came before.

But I suspect -and keep in mind that I wasn't there- that for DNSSEC hatred of ASN.1 was only part of it, with the other part being a desire for proper name constraints[0] and trust hierarchy[1].

It does make sense that if the names we wish to authenticate are hierarchical in nature[2] then the PKI for it should use the same hierarchy. The only reason that the WebPKI isn't structured the same as DNSSEC is that historically the support for x.509 naming constraints wasn't there, then the business model of the WebPKI made all those on the inside not want to fix that. Now that we have Let's Encrypt the latter problem is going away, but the underlying issues remain.

This is where you and I part and disagree. You greatly dislike DNSSEC, while I greatly like it. You especially dislike DANE, and I especially want DANE. But it's nice to know that we at least agree on ASN.1 and x.509.

[0] x.509 has this, but it's ill-supported in WebPKI land.

[1] x.509 also has, kind of[3] but which, again, WebPKI does not.

[2] DNS naming is strictly hierarchical, though one can be forgiven for thinking that the com. zone is just one big flat swamp as it really is, but it's not the whole of the DNS.

[3] Well, in x.509 the hierarchy of CAs is based on x.500 naming, which is just not appropriate for a web where URI authority names are DNS names (to say nothing of what a disaster x.500 naming is). There's alternative issuer names, which includes dNSName, but once more, the issue is how widespread the support for those is.

EDI, PDF, and ACPI DSDT AML are provably worse "standards" as "standards" go.

Forking implies a problem in one or more areas:

- governance

- usability (documentation and examples)

- potentially NIH-syndrome

protobufs, capnproto, flatbuffers, thrift, msgpack, erlang terms, BSON, java class files, and more exist to scratch itches that are not universally scratchable. You can't argue with the readability, portability, simplicity, or usability of JSON.. although you can argue with it's low density and parsing cost.

What's missing generally in computer science, software engineering, and systems engineering is standardization down to a minimal, feature-complete subset of interchange formats, protocols, and APIs in the spirit of professionalizing and standardizing the industry with trusted conventions. The issue has been raised many times over the past 60 years, but at some point, there needs to be more than the rare bird of "professional engineering" licensure and too common and useless A+ certifications.

I like DNSSEC and DANE because the combination adds another security layer to corroborate what PKI says. If browsers adopted it, it would be widely deployed to web-facing services. Like GPG, it has a UX problem and is a PITA to setup. (Sadly, a major DNSSEC multi browser extension effort shutdown.) I thought we were pro defense-in-depth rather than less security, so I don't understand the hate DNSSEC receives because it's not an alternative so much as a secondary trust-supporting mechanism.

Disclaimer: I run PKI at home where every machine has 802.11x certs, every "intranet" has a cert from an intermediate cert, and every service like vSphere has its own subordinate signing cert. I keep meaning to migrate over to FreeIPA with Dogtag.

It effectively adds another CA to the set of CAs we already have, but this time without any of the transparency logs or browser leverage to control abuse. When operators screw up WebPKI certificates, people get certificate errors in their browser. When they screw up DNSSEC --- which everybody does --- their whole domain falls off the Internet. It doesn't solve any problems, and creates new ones. It's easy to understand the hate DNSSEC gets.

Shall we reprise our arguments about this again for GP's benefit?

Nothing about DNSSEC precludes the addition of CT, and it will eventually happen.

DNSSEC w/ query name minimization means that any zone that wishes to MITM you has to potentially make the decision without first knowing what your target domain is, or even who you are (obviously this last depends on context), and the more indiscriminate they are in their MITM activities the more likely that they'll be caught. That's a big deal, and something WebPKI can't offer -- WebPKI only has CT to offer for mitigating the MITM power of CAs, and CT depends on people auditing the logs.

Only any entity in the world that can effectively require the major TLDs to implement DNSPKI transparency. The browsers forced CT on the CAs, under threat of removing them from the root programs.

> You can't argue with the readability, portability, simplicity, or usability of JSON.

Hah, the joke's on you: ASN.1 has JER, which is the ASN.1 JSON Encoding Rules :)

So you can specify JSON schemas in ASN.1 syntax, which means that you could transcode from any of the other ASN.1 encoding rules, which include:

  - BER/DER/CER (TLV encodings)
  - OER/PER (like XDR, but with one octet alignment)
  - XER (yes, the X is for XML -- XML Encoding Rules)
  - GSER (a pre-JSON JSON-like encoding rules for
    ASN.1 invented at IETF)
  - and other ERs that are proprietary
  - and any ER you care to create, which could just
    be bindings of other encoding schemes like
    Flat Buffers or anything else you like.

    I've been wanting an XDR-based encoding rules...

Which goes to show that a lot of the encoding schemes you listed could just have been ASN.1 ERs from the get-go and:

  - benefited from existing ASN.1 modules
  - benefited from transcoding
  - benefited from existing ASN.1 compiler codebases

But no, people had to reinvent the wheel all the way to the syntax and semantics without adding much of anything truly new.

> I like DNSSEC and DANE because the combination adds another security layer to corroborate what PKI says. If browsers adopted it, it would be widely deployed to web-facing services. Like GPG, it has a UX problem and is a PITA to setup. (Sadly, a major DNSSEC multi browser extension effort shutdown.) I thought we were pro defense-in-depth rather than less security, so I don't understand the hate DNSSEC receives because it's not an alternative so much as a secondary trust-supporting mechanism.

I agree with this, though I'm not sure what is the antecedent of "it" in "[l]ike GPG, it has a UX problem..." -- could be DNSSEC, but it could also be PKI! :)

Seems most counterproductive if you lose.

DJB is a gift to humanity.

As if djb not being an asshole would have saved the IETF from their incompetent buffoonery. Yeah let’s just blame it all on one guy for not being a sycophant rather than lay the responsibility on the actual buffoons.

Other than source-less reputation bashing / gossip, are instances of poor behavior?

Also consider TdR of OpenBSD and OpenSSH has been polarizing but impactful.

I think the track records are tellingly comparable.

Sounds like Theo de Raadt.

Except that a huge amount of the world runs on cryptography designed by him, because all the standards committees and government orgs are somehow less trustworthy than this "asshole".

djb is like a litmus test.

How someone reacts to him can reveal whether that person is someone who thinks they are smarter than they really are, a personality type that is common amongst software developers. These are the type of people who cannot admit when they are wrong, and will never admit that they are incompetent. Arguably, it is because of such people that we are drowning in crappy software, and even worse, that crappy software is popular.

IMHO, djb is on the side of the user, whether intentionally or not. djbdns allowed me, a user, to take control of DNS on the networks and computers I own. The philosophy of BIND at the time felt like "DNS is for DNS experts. We're experts. Do as we tell you."^1 I also learned nsd so I do use BIND zone file format. But I never became a BIND user.

1. I see a parallel in the context of "certificate authorities". Nothing wrong with "CAs" in theory but in practice, i.e., commercialisation of certificates and browser-centricity, the system reeks. Pay to play, or at least mandatory third party intermediaries, and generally user hostile.

I think it might be dangerous to elevate people in this way. Probably djb is a human, who doesn’t represent a dowsing rod for your evaluation of the world.

only this one instance? what about port randomization or Curve25519?

Like everyone else, he's right sometimes and wrong others.

The two examples above are cases where he was correct actually.

See previous statement.

I think it's good that there's a standard for zone files. I've migrated from one dns server to another and being able to just carry over the same zone files has been useful. I also for some time had a master using one dns software and slaves using different software (as part of a mirgariton keeping a known good server using the old software and moving others to the new software and testing to ensure both were producing the same responses). Had zone transfers not been standardised I would not have been able to do this. I agree that perhaps the easy path of just adoptong the BIND implementation should not have happened but I would argue having a standard is better than just leaving it to hoping the implementation detail remains compatible between different software.

It's absolutely useful to have a standard form for DNS zone files; it just shouldn't be a part of the DNS standard. There's no scarcity of RFC numbers, and zone files should be a separate, optional RFC.

(They're practically obsolete now, so there isn't much uncertainty about who had the better side of this argument.)

Not sure what you mean by obsolete.

https://www.internic.net/domain/root.zone

I agree that they are optional, but I wouldn't go so far to say they're obsolete.

Anybody building a new DNS server at this point is going to back it with a database, not a bunch of text files.

PowerDNS [1] has been doing that under the GPL since 2002, featuring a large number of different backends ranging from simple BIND style zonefiles to relational databases and load balancing/failover algorithms.

[1] https://www.powerdns.com/

> I'd argue that even AXFR (zone transfers) as a replication mechanism should have been a BIND implementation detail.

AXFR has been valuable for interop, though, and allows you to have diverse implementations of authoritative servers.

The syntax of zone files originally came from JEEVES, not BIND. It’s implemented by most other DNS software, and is not at all specific to BIND. Same for AXFR and other parts of the DNS protocol.

I read this and if I understand it, someone wants to add some kind of automatic key exchange kind of like what happens with 'https:' ?

Please do not do this :)

One comment I agree with from a linked article:

https://mjg59.dreamwidth.org/65874.html

>It takes all the problems associated with HTTPS and makes it SSH's problem also. This relates also to the reasons why OpenSSH refuses to accept patches that implement SSL/TLS pki and instead implemented their own CA model.

No, I'd like sites to be able to present a CA to trust rather than a static key to trust and then be able to rotate their keys without triggering identity change notices.

This stuff is a bit over my head.

Under this scenario, if the site rotates their key, will I need to decide on my system to trust it or not ? Maybe change something on my system ?

Or does the remote site use my ssh public key transparently to know it is me, and allow me in ? So I do nothing at my end ? I guess that would be OK by me. But as I noted above, I am guessing.

Your key is not involved here at all.

You can also sign host keys and then trust the CA instead of the host key with TOFU, but as mentioned in the article, there's no way to do this without having the user fiddle with their SSH config and I'm unsure how good the tooling for distributing revocations is.

Vault has a setup guide, along with the little dance a user would have to do: https://developer.hashicorp.com/vault/docs/secrets/ssh/signe...

Under this scenario you wouldn't need to do anything if the site rotates their key - the new key would be presented with a certificate signed with the same CA key. But no clients support doing this kind of semi-automated enrollment at the moment. You can manually add the trusted certificate to known_hosts, though.

Great, thanks

The way I understand SSH interaction, is that every single time I login to a site or server for the first time, it is like interacting with self-signed HTTPS cert. There is no PKI, no central trust that did any kind of validation that the site is who they say they are.

Considering that SSH is generally used by a small number of people per server vs. a general access mechanism (for the general internet), the need for PKI hasn't seemed as pressing. But why it isn't more prominent, I don't know.

If I understand it correctly, there would be zero change for the user.

Right now, a server has a single host key. This is generated once and will remain static for the server's entire lifetime. The client has the public part of that in their authorized_keys. When a session is created, the server uses the private part to sign some stuff, and the client validates it using the already-known public key. In other words, we have the host key K, client stores K_public in known_hosts. On session creation, server does `session_signature = K_private.sign(data)` and sends (data, session_signature) to client. Client does `K_public.validate(session_signature, data)` and continues if it all matches.

This usually works just fine, but you can only have a single key for a server - which is a bit of an issue if that "server" is actually an entire cluster consisting of thousands of machines. Every singly machine needs to have a copy of the single private key. as delegating this to a secure enclave is unfeasible if you get hundreds of thousands of connections per second. Not to mention that key rotation is a bit of a nightmare.

With this proposal, we add a parent key, let's call it L. L_private is stored in some kind of very secure hardware enclave. Every hour/day/whatever, the server generates a new host key K, and asks the enclave to sign the new public key. The server now has K_private, and `host_signature = L_private.sign(K_public)` as generated by the enclave. The client stores L_public in their known_hosts. On session creation, server does `session_signature = K_private.sign(data)` and sends (data, session_signature, K_public, host_signature) to the client. The client finally does `L_public.validate(host_signature, K_public) && K_public.validate(session_signature, data)`.

All of this happens on the server side, and the only thing needed is a relatively minor protocol upgrade. The interesting part is that a very similar mechanism already exists for the client's key, making it way easier to manage hundreds of keys across thousands of servers without going crazy by having to update authorized_keys among the entire server fleet multiple times per week.

It would be better at an intermediate cert level. Those are generally more aligned to an organization's boundaries.

The format doesn't support intermediates.

What about a way to sign the new key with the old key?

If the host presents a new key, plus a signature from the old key which you already trusted, then your system switches to trusting the new key?

I can see that there are problems in that if you don't attempt to log in between when the new key is issued and the old key expires, then you won't be able to trust the switch and will have to re-authenticate via external means. That might be a problem, but it might be easier for some groups of people to manage than figuring out a whole new CA model?

That's fine for orderly transitions, but if you're worried that your private host keys have been compromised then (a) you can't trust any new transitions and (b) if the new private keys are already on the systems then have they also been compromised?

However, we don't need to have the new private keys in order to tell people about the new public keys using our old private keys.

If a valid emergency private key lives in the safe, realistic bad guys (who don't do black bag jobs, they just read bug lists and act quickly) can't have that key until you put it into use.

So you could have a situation where say, you schedule key replacement every 12 months, you always immediately generate and announce the next key, but it lives in the safe until the rotation happens or something goes wrong and you use it immediately.

The automatic transition would not cause additional harm in the compromise scenario and would in fact marginally help. A client which has not got an out-of-band revocation/update of the host key is subject to complete mitm'ing and impersonation either way: tricking it into accepting a new fraudulent host key doesn't actually make this scenario worse, it only means the first attacker locks out his competition :). (An out-of-band key update still fixes the client.) OTOH, if the client happens to connect to the authentic host first, then it is saved. So, no net harm, marginal net benefit.

SSH certificates already enable this. I use them in prod.

> I read this and if I understand it, someone wants to add some kind of automatic key exchange kind of like what happens with 'https:' ?

SSH already does an automatic key exchange when you start a session (and has for decades); the private/public keys used for authentication aren't the same keys as used for encryption of an established session.

Dropbear seems pretty common on busybox devices (or at least consumer routers) https://matt.ucc.asn.au/dropbear/dropbear.html

Dropbear is also very portable to lots of architectures and OSes, and runs well on small systems. I endorse it highly.

It's the 2nd most popular SSH daemon across the Internet: https://www.shodan.io/search/facet?query=port%3A22&facet=pro...

OpenSSH is indeed widely used. But the only game in town they're not. There are commercial SSH implementations that aren't based on OpenSSH, reasonably widely used too. For decades I've used a Windows product called SecureCRT based on their own ssh and sshd stacks at Van Dyke. I'm sure there are others too. (I hated putty's UI.)

There's an obvious infosec observation here. Different stacks, different hacks. Maybe individual infosec people should adopt a randomly chosen stack from among the less popular. Then the dreaded zero-day that hits the most popular stack won't cut off their access entirely.

I totally forgot that the very original version of SSH came out in 1995 and turned into a commercial product. https://www.ssh.com/home-page

We run it on our VAX.

The article also says:

"I don't think anyone else is working on new protocol features; instead, OpenSSH comes up with them and then people with other SSH implementations either follow along or not."

One obvious exception is curve25519-sha256@libssh.org which is the preferred kex.

To this day I occasionally attempt to run `emerge ssh` to update it and get back portage's version of "I can't do that, Dave". Then, "oh, right..." `emerge openssh`.

I'm curious to know what are the advantages of using those other implementations instead of OpenSSH. Was it mostly because Windows didn't have openssh? Or do they provide more features (or less! which could be good for security)? I have only ever used OpenSSH so I'm totally ignorant about everything else!

I went with SecureCRT because it had its own well-done terminal emulator back in the mid-1990s when I started using it. It's evolved nicely ever since.

Case studies of OpenSSH vulnerabilities are fairly limited, iirc there are a few ssh engines with fairly frequent issues.

Safety in numbers is probably a better option than security through obscurity in this case.

Another way I'm feeling old... I remember back when OpenSSH was just a plucky upstart implementation of SSH.

I did the original import of the unencumbered SSH sources into the OpenBSD CVS repository (imported to the wrong path and Theo blew up lol). Haven't thought about it in ages and the distance of the memory is making me feel old too.

>theo blew up

Those rash of OpenSSH and OpenSSL vulnerabilities back in the late 90s seemed like it was targeted at Theo just because he acted like a jerk and he knew it. The never compromised tagline of OpenBSD was flushed down the toilet and at the time it really seemed like it was a deliberate attack

Note that OpenSSL is not an OpenBSD project.

LibreSSL is.

If you bombastically claim something then don't be surprised if people are going to test that claim.

Yep. I wanted to say it was F-Secure SSH, but the timeline on Wikipedia doesn't quite match. The development of SSH and the history of Data Fellows/F-Secure kinda line up, though.

https://en.wikipedia.org/wiki/Secure_Shell https://en.wikipedia.org/wiki/F-Secure

Update: yep, it was DataFellows that I remember! https://www.openssh.com/history.html

> Later licenses restricted the use of ssh in a commercial environment, instead requiring companies to buy an expensive version from a company called Datafellows.

Yup, the good ol' days of DataFellows, when I had to convince people to use SSH instead of Telnet... ;-)

OpenSSH’s relationship to SSH Labs is similar to what Linux is to Unix: one is considered the default implementation (where you like it or not) and the other an irrelevant standard (likewise).

Or Chromes relationship to Web :)

This makes me think about something I thought of a while back, is there any library/app which lets me expose a TUI/cli-app over ssh without actually caring about securing OpenSSH for this use case?

Are you thinking of something like the Blinkenlights Star Wars telnet demo from a while ago?

The "securing" part aside, ssh still gives you a whole terminal experience very similar to the telnet one & the ability to draw into it (like if you were using Emacs on the shell host).

You can set something like that up by forcing a command on ssh instead of a shell (like what github does for git).

Yeah I was thinking of something like that. But I still feel a bit uneasy with configuring OpenSSH securely, I’ve escaped such setups before, gaining full shell access through exploits. Maybe I’m overly paranoid.

No you’re not, there are far too many unexplored interfaces.

Then again many programs aren’t secure against untrusted input either.

Like ForceCommand, or turning off authentication?

Do you mean something like Paramiko, an SSH library for Python?

https://www.paramiko.org

https://github.com/charmbracelet/wish

If you're actually willing to write the app.

Something like Apache Guacamole as a https web to SSH "something" gateway?

Go to http-something in your web browser which is a Guacamole install and log in to Guacamole as "guest" / "guest" then in guac the only connection option is some SSH login that'll appear in your web browser

As a warning people have tried to supply "nethack as-a-service" or whatever and people are really creative about gaining shell access somehow anyway, via buffer overflows or who knows what. You can do it, just be careful.

Maybe "poor man's SSH via websocketd" could be a building block?

https://github.com/rowanthorpe/ws-repl

Tailscale comes to mind, see https://tailscale.com/kb/1100/services/

someone posted an ssh bbs the other week https://news.ycombinator.com/item?id=35412416

The go ssh packages are a popular choice:

https://github.com/shazow/ssh-chat

https://github.com/zachlatta/sshtron

https://github.com/quackduck/devzat

https://github.com/donuts-are-good/shhhbb/

There's also a assembly library:

https://2ton.com.au/sshtalk/

And for rust there's trush, and for python paramiko as mentioned.

> expose a TUI/cli-app over ssh without actually caring about securing OpenSSH

If you already have an app, see maybe:

https://drewdevault.com/2019/09/02/Interactive-SSH-programs....

Hello, I made devzat. I can try to answer if anyone has any questions about ssh-based apps

> And for rust there's trush

Do you mean thrussh?

It looks like that hasn't been updated in 2 years. But there is an active fork of it called russh.

Not exactly what you're looking for but to solve a similar problem I've been working on a TUI-over-the-wire protocol.

http://uggly.bytester.net

Interestingly enough this is getting another layer on top of it. Google is working on something called "HIBA" (https://github.com/google/hiba) which is an extension to OpenSSH SSH Certificates.

So now you have layers of:

- SSH RFC

- OpenSSH extensions to SSH

- Google extensions to OpenSSH

This looks like it's adding something similar to Kerberos-like authentication the connection rather than the OS auth level. Pretty cool.

More layers, more abstractions, more divergence, more interoperability problems, more vulnerabilities.

There's already mature CA support in OpenSSH

https://dev.to/gvelrajan/how-to-configure-and-setup-ssh-cert...

And even OpenSSH 8.2+ supports FIDO2 built-in.

Can add TOTP, FIDO2/u2f, ldap, or kerb with pam also.

What’s the value of a CA in openSSH? Is it purely so you don’t have to self manage known_hosts, based on mutual trust with the CA?

Edit: or is there some ability for clients to leverage the CA to establish trusts without authorized_keys?

Both. Either. See more: https://web.archive.org/web/20201203110622/https://ef.gy/har...

I feel OpenSSH is more powerful overall. Its cryptographic key management allows users to generate and manage keys securely which useful for larger organisations, where managing a large number of keys can be challenging.

Just to make sure. OpenSSL is not related to OpenSSH, right

Is there a citation for this:

> In one recently relevant example, one reason Github didn't take advantage of OpenSSH's protocol extension for offering multiple host keys (to enable upgrades or transitions) is that they don't use OpenSSH but instead a different implementation.

? I thought GitHub didn't use the host-key-migration capability because the host key change was implemented on a semi-emergency basis. I mean, in this case I don't think you'd want to start a host-key migration without announcing it, and GitHub wanted off of the exposed RSA host key very quickly.

Reply on news.ycombinator.com