OpenSSH introduces options to penalize undesirable behavior
undeadly.org
undeadly.org
From my experiments with several honeypots over a longer period of time, most of these attacks are dumb dictionary attacks. Unless you are using default everything (user, port, password), these attacks don't represent a significant threat and more targeted attacks won't be caught by this. (Please use SSH keys.)
I have seen experienced sysadmins create the test user with the password of "test" on a live server on port 22 because they were having an "autopilot moment". It got hacked within 20 minutes of going online and these mechanisms wouldn't have saved it, the attacker got in on the second or third try.
If you want to have a read about unsolved problems around SSH that should be addressed, Tatu Ylonen (the inventor of SSH) has written a paper about it in 2019: https://helda.helsinki.fi/server/api/core/bitstreams/471f0ff...
I'd say "well, it might catch the lowest effort attacks", but when SSH keys exist and solve many more problems in a much better way, it really does feel pointless.
Maybe in an era where USB keys weren't so trivial, I'd buy the argument of "what if I need to access from another machine", but if you really worry about that, put your (password protected) keys on a USB stick and shove it in your wallet or on your keyring or whatever. (Are there security concerns there? Of course, but no more than typing your password in on some random machine.)
The problem is that you can issue keys without having to deploy them to a fleet of servers (you sign the user's pubkey using your SSH CA key), but you have no way of revoking them without pushing an updated revocation list to the whole fleet. We did have a few long-term keys that were issued, generally for build machines and dev environments, and had a procedure in place to push CRLs if necessary, but luckily we didn't ever end up in a situation where we had to use it.
Fun fact: it was just a few months ago that Heimdall Kerberos started respecting CRLs at all, that was a crazy bug to discover
see https://man.openbsd.org/ssh-keygen.1#KEY_REVOCATION_LISTS for more info
And unlike some other sshd directives that have a 'Command' alternative to specify a command to run instead of reading a file, this one doesn't, so you can't just DIY distribution by having it curl a shared revocation list.
[0] OpenPubkey - https://github.com/openpubkey/openpubkey/
[1] SSH3 - https://github.com/francoismichel/ssh3
With a fixed expiration, if you choose a 2 hour expiry, the user has to reauth every 2 hours each time they start a new SSH session.
With a refreshable expiration, if you choose a 2 hour expiry, the user can refresh the certificate if they are still logged in.
This lets you set shorter expiry times because the refresh token can be used in the background.
pam_pwnd[1], testing passwords against the Pwned Passwords database, is a(n unfortunately abandoned but credibly feature complete) thing. (It uses the HTTP service, though, not a local dump.)
if the password have decent entropy, it won't be in the top 1000 of the leaks so not used in blond brute force like this.
I’m sure this will be fixed by just telling everyone to disable IPv6, par for the course.
So instead of looking, like the author of these new options, for ways to make life for the bad guys harder we do nothing?
Your concerned are addressed in TFA:
> ... and to shield specific clients from penalty
> A PerSourcePenaltyExemptList option allows certain address ranges to be exempt from all penalties.
It's easy for the original owner to find the list of all the IP blocks the three or four ISPs he's legitimately be connecting from to that exemption list.
I don't buy your argument nor all the variation on the same theme: "There's a minuscule risk of X, so we absolutely nothing but saying there's nothing to do and we let bad guys roam free!".
There's nothing more depressing than that approach.
Kudos to the author of that new functionality: there may be issues, it may not be the panacea, but at least he's trying.
The thing is, we have tools to implement this without changing sshd's behavior. `fail2ban` et. al. exist for a reason.
This is exactly the sort of thing that should be part of the server. In exactly the same way that some protocol clients have waits between retries to avoid artificial rate limiting from the server.
There are a lot of ways a builtin facility of one service can go wrong, especially if it ends up being active by default on a distro.
`fail2ban` is common, well known, battle-tested. And its also [not without alternatives][1].
Also adding firewalling to SSH is hardly “kitchen sinking” (as another commenter described it). You’re literally just adding another layer of security into something that’s literally meant to be used as an out of the box solution for creating secure connections.
If you want to take issue with the “kitchen sink” mentality of SSH then complain about its file transfer features or SOCKS support. They are arguably better examples of feature creep than literally just having the server own what connections it should allow.
sshd is a service. It may be one among dozens of other services running on a host.
Now imagine for a moment, if EVERY service on the host took that approach. Every backend service, every network-facing daemon, every database, every webserver, voip servers, networked logging engines, userspace network file systems, fileservers...they all now take security into their own hands.
Every single one of them has its own fail2ban-ish mechanism, blocklists it manages, rules for what to block and how long, what triggers a block, if and when a block will be lifted...
Oh, and of course, there is still also a firewall and other centralized systems in place, on top of all that.
How fun would such a system be to administer do you think? As someone with sysadmin experience, I can confidently say that I would rather join an arctic expedition than take care of that mess.
There is a REASON why we have things like WAFs and IDS, instead of building outward-facing-security directly into every single webservice.
That was additional initial effort but the change made sense and we sysadmins coped fine.
Likewise, there was a time when server side website code had to be invoked via a httpd plugin or CGI, now every programming language will have several different web frameworks, each with their own HTTP listener and each needing to be configured in its own unique way.
Like with inetd, the change made sense and we managed just fine.
Tech evolves — it’s your job as a sysadmin to deal with it.
Plus, if you’re operating at an enterprise level where you need a holistic view of traffic and firewalling across different distinct services then you’d disable this. It’s not a requirement to have it enabled. A point you keep ignoring.
And still we keep all those webservices, be they in Java, Go, node, C# or Python, behind dedicated webservers like nginx or apache.
Why? Because we trust them, and they provide a Single-Point-Of-Entry.
> Tech evolves — it’s your job as a sysadmin to deal with it.
Single-Point-Of-Entry is still prefered over having to deal with a bag of cats of different services each having their own ideas about how security should be managed. And when a single point of entry exists, it makes sense to focus security there as well.
This has nothing to do with evolving tech, this is simple architectural logic.
And the first of these points that every server has, is the kernels packet filter. Which is exactly what tools like fail2ban manage.
> A point you keep ignoring.
Not really. Of course an admin should deactivate svc-individual security in such a scenario, and I never stated otherwise.
The point is: That's one more thing that can go wrong.
Not really no. They might sit behind a load balancer but that's to support a different feature entirely. Some services might still be invoked via nginx or apache (though the latter has fallen out of fashion in recent years) if nginx has a better threading model. But even there, that's the exception rather than the norm. Quite often those services will be stand alone and any reverse proxying is just to support orchestration (eg K8s) or load balancing.
> Single-Point-Of-Entry is still prefered over having to deal with a bag of cats of different services each having their own ideas about how security should be managed.
Actually no. What you're describing is the castle-and-Moat architecture and that's the old way of managing internal services. These days it's all about zero-trust.
https://www.cloudflare.com/en-gb/learning/security/glossary/...
But again, we're talking enterprise level hardening there and I suspect this openssh change is more aimed at hobbyists running things like Linux VPS
> > A point you keep ignoring.
> Not really. Of course an admin should deactivate svc-individual security in such a scenario, and I never stated otherwise. The point is: That's one more thing that can go wrong.
The fact that you keep saying that _is_ missing the point. This is one more thing that can harden the default security of openssh.
In security, it's not about all or nothing. It's a percentages game. You choose a security posture based on your the level of risk you're willing to accept. For enterprise, that will be using an IDP to manage auth (including but not specific to SSH). A good IDP can be configured to accept requests from non-blacklisted IP, eg IPs from countries where employees are known not to work in), and even only accept logins from managed devices like corporate laptops. But someone running a VPS for their own Minecraft server, or something less wholesome like Bit-Torrent, aren't usually the type to invest in a plethora of security tools. They might not even have heard of fail2ban, denyhosts, and so on. So having openssh support auto-blacklisting on those servers is a good thing. Not just for the VPS owners but us too because it reduces the number of spam and bot servers.
If your only concern is that professional / enterprise users might forget to disable it, as seems to be your argument here, then it's an extremely weak argument to make given you get paid to know this stuff and hobbyists don't.
I would trust the OpenSSH developers to do a better job with the much simpler requirements associated with handling it within their own software.
https://alanj.medium.com/do-one-thing-and-do-it-well-a-unix-...
It's battle-tested for well over a decade, has accumulated 10.8k stars and 1.2k forks on github, so it seems to do something right no?
Not to mention that even if it were otherwise, that's not a reason to ignore UNIX philosopies that have served the FOSS world well for over half a century at this point.
Last but not least, there are any number of alternative solutions.
It's useful because you can hook it up to anything that produces logs, it's hacky because that means you are using regexp. If the log format changes, you're likely fucked, not to mention that regexps are notoriously hard to make 'air tight' and often screwed up by newbies. Add to that in a case where your regexes start missing fail2ban will stop doing it's job silently.. not great my friend.
It's been a useful hack for a very long time, but I'd like to see us move on from it.
It doesn’t hurt to have this functionality in openssh too. If you still need to use fail2ban, denyhosts, or whatever, then don’t enable the openssh behaviour feature. It’s really that simple.
UNIX Philosophy: "Do one thing, and do it well". An encrypted remote shell protocol server should not be responsible for fending off attackers. That's the job of IDS and IPS daemons.
Password-based ssh is an anachronism anyway. For an internet-facing server, people should REALLY use ssh keys instead (and preferably use a non-standard port, and maybe even port knocking).
If you have a IDS and IPS set up then you’re already enterprise enough that you want your logs shipped and managed by a single pane of glass. This new SSH feature isn’t intended to solve enterprise-level problems.
Plus if you want to argue about “unix philosophy” with regards to SSH then why aren’t you kicking off about SOCKS, file transfer, port forwarding, and the countless other features SSH has that aren’t related to “shell” part of SSH? The change you’re moaning about has more relevance than most of the other extended features people love SSH for.
But service level security features have the potential to cause enterprise-level problems.
Sure, in an ideal world, all admins would always make zero mistakes. And so would the admins of all of our clients, and their interns, and their automated deployment scripts. Also in that perfect world, service level security features would never be on by default, have the same default configuration across all distros, and be easy to configure.
But, alas, we don't live in a perfect world. And so I have seen more than one service-level security feature, implemented with the best of intentions, causing a production system to grind to a halt.
Only if you don’t know what you’re doing. Which you should given you’re paid to work on enterprise systems.
Whereas not having this causes problems for users are not paid to learn this technology.
So it seems completely reasonable to tailor some features to lesser experienced owners given the wide spectrum of users that run openssh.
Picture the amount of Captchas you already getting from a legitimate Chrome instance, but instead of by-passable annoying captchas, you are just locked out.
After this party, this guy needed help, he lost his wallet and his phone, his sister also went to the party and gave him a ride there but had left. he didn't know her number to call her, and she'd locked down her socials so we couldn't use my phone to contact her. we were lucky that his socials weren't super locked down and managed to find someone that way, but priv keys are only good so long as you have them.
I have a backup key in storage. I have escrow mechanisms. These would be inconvenient, but, it's been 40 years since I've lost any keys or my wallet, so I feel pretty good about my odds.
Which is what the game here is. The odds. Famously humans do poorly when it comes to this.
My ssh keys are encrypted. They need a password, or they are worthless.
Sure, I can mistype that password as well, but doing so has no effect on the remote system, as the ssh client already fails locally.
Doesn’t change the advice, if this is your only management interface, don’t enable it :)
Also you know you can have MFA even with pw authentication right? :)
And that's still assuming you end up in a range that is actively attacking your sshd. It's definitely possible but really doesn't seem like a bad tradeoff
Unless you're only blocking connection attempts to ssh and not the VPN, but what good is that? There is no reason to expect the VPN to be any more secure than OpenSSH.
Hacking into the VPN doesn't get the attacker into the SSH server too, so there's defense in depth, if your concern is that sshd might have a vulnerability that can be exploited with repeated attempts. If your concern is that your keys might be stolen, this feature doesn't make sense to begin with.
Websites usually don't care about ssh brute force attempts because they don't listen on ssh. But the issue isn't websites anyway. The problem is that your server is blocking you, regardless of what websites are doing.
> If that doesn't matter to you and all you need is to get into a particular SSH server, and also the network blocks VPNs, you're still fine if the SSH is on port 9022 and VPN is port 22. If it's not your own SSH server and it's port 22, then you're still fine if your own VPN is port 22 (on a different host).
Then you have a VPN exposed to the internet in addition to SSH, and if you're not rate limiting connections to that then you should be just as concerned that the VPN "might have a vulnerability that can be exploited with repeated attempts." Whereas if the SSH server is only accessible via the VPN then having the SSH server rate limiting anything is only going to give you the opportunity to lock yourself out through fat fingering or a misconfigured script, since nobody else can access it.
Also notably, the most sensible way to run a VPN over TCP port 22 is generally to use the VPN which is built into OpenSSH. But now this change would have you getting locked out of the VPN too.
Things that could be done is making password auth harder to configure to encourage key use instead, or invest time into making SSH CAs less of a pain to use. (See the linked paper, it's not a long read.)
Random brute force attempts against SSH are already a 100% solved problem, so doing nothing beyond maintaining the status quo seems pretty reasonable IMO.
> I don't buy your argument nor all the variation on the same theme: "There's a minuscule risk of X, so we absolutely nothing but saying there's nothing to do and we let bad guys roam free!".
Setting this up by default (as is being proposed) would definitely break a lot of existing use cases. The only risk that is minuscule here is the risk from not making this change.
I don't see any particularly reason to applaud making software worse just because someone is "trying".
Yes, because as soon as the security clowns find out about these features, we have to start turning it on to check their clown boxes.
OpenSSH already seems to take that into account by allowing you to penalize not just a single IP, but also an entire subnet. Enable that to penalize an entire /64 for IPv6, and you're in pretty much the same scenario as "single IPv4 address".
I think there's some limited value in it. It could be a neat alternative to allowlisting your own IP which doesn't completely block you from accessing it from other locations. Block larger subnets at once if you don't care about access from residential connections, and it would act as a very basic filter to make annoying attacks stop. Not providing any real security, but at least you're not spending any CPU cycles on them.
On the other hand, I can definitely see CGNAT resulting in accidental or intentional lockouts for the real owner. Enabling it by default on all installations probably isn't the best choice.
Or better yet, a corp VPN with good security practices so you don't need this fail2ban-type setup. It's also weird to connect from home using password-based SSH in the first place.
Sometimes it's not a matter of price. If you're traveling your only option for a network connection could be whatever dreck the hotel deigns to provide.
Also it's unlikely that your hotel's IP address is spamming the particular SSH server you need to connect to.
DoS in this context is generally pretty boring. Your CPU would end up at 100% and the service would be slower to respond but still would. Also, responding to a DoS attempt by blocking access is a DoS vector for anyone who can share or spoof your IP address, so that seems like a bad idea.
If someone is trying to exploit sshd, they'll typically do it on the first attempt and this does nothing.
> Also it's unlikely that your hotel's IP address is spamming the particular SSH server you need to connect to.
It is when the hotel is using the cheapest available ISP with CGNAT.
If the hotel is using such a dirty shared IP that it's also being used to spam random SSH servers, that connection is probably impractical for several other reasons, e.g. flagged on Cloudflare. At that point I'd go straight to a VPN or hotspot.
And there is no guarantee you can use another network connection. Hotspots only work if there's coverage.
Plus, "just use a hotspot or a VPN" assumes you were expecting the problem. This change is going to catch a lot of people out because the first time they realize it exists is during the emergency when they try to remote in.
That may not be an option at all, especially with working from home or while traveling.
For example at my home all ISPs i have available use cgnat.
Your work doesn't provide a VPN?
> For example at my home all ISPs i have available use cgnat.
Doubtful - you probably just need to pay for a business line. Somtimes you can also just ask nicely for a non-NATed IP but I imagine this will get rarer as IP prices increase.
Build tools that follow the standard/best practices by default, maybe build in an exception list/mechanism.
IPv6 space is plentiful and easy to obtain, people who are allocating it incorrectly should feel the pain of that decision.
People should write 80/48 or 48/80 to be clear
> People should write 80/48 or 48/80 to be clear
The clarity is found implied in your preferred example.
- "80/" would mean "80 bits before"
- "/48" would mean "48 bits after"
If you have a /48 then 48 bits are used to determine the address is yours. Any address which matches in the first 48 bits is yours. If you have a /64, any address which matches in the first 64 bits is yours.
99/100 it means the first bits.
$ sipcalc ::/48
-[ipv6 : ::/48] - 0
[IPV6 INFO]
Expanded Address - 0000:0000:0000:0000:0000:0000:0000:0000
Compressed address - ::
Subnet prefix (masked) - 0:0:0:0:0:0:0:0/48
Address ID (masked) - 0:0:0:0:0:0:0:0/48
Prefix address - ffff:ffff:ffff:0:0:0:0:0
Prefix length - 48
Address type - Reserved
Comment - Unspecified
Network range - 0000:0000:0000:0000:0000:0000:0000:0000 -
0000:0000:0000:ffff:ffff:ffff:ffff:ffff
I recently found a case where an "inverted" netmask makes sense - when you want to allow access through a firewall to a given IPv6 host (with auto-config address) regardless of the network that your provider has assigned.
Then I started noticing that the database was randomly "getting stuck" on the test system. This went on for a few times until I noticed that I exposed the database to the internet with postgres/postgres as credentials.
It might have been even some "friendly" attackers that changed the password when they were able to log in, to protect the server, maybe even the hosting provider. I should totally try that again once and observe what commands the attackers actually run. A bad actor probably wouldn't change the password, to stay unnoticed.
I've faced the DMZ at an IP on DHCP. Later when the host changed I had noticed traffic from the internet getting blocked on the new host and realized my mistake.
Just silencing all the failed attempts may be better. So much noise in these logs anyway.
Unfortunately there's no other way. Security always goes back to economics; you must make the abuse cost more than it's worth. Phone-based 2FA is also an anti-spam measure, cause clean phone numbers cost $. When trying to purchase sketchy proxies or VPNs, it basically costs more to have a cleaner ip.
My password is **********
See: it works! Try it!
An SSH key can be freely reused to log in to multiple SSH servers without compromise. Passwords should never be reused between multiple servers, because the other end could log it.
An SSH key can be stored in an agent, which provides some minor security benefits, and more importantly, adds a whole lot of convenience.
An SSH key can be tied to a Yubikey out of the box, providing strong 2FA.
Password authentication, which only uses your password to establish you are authentically you, does not establish the same level of cryptographic trust, and also does not allow the SSH server to bail out as quickly, instead needing to perform more crypto operations to discover that an unauthorized authentication attempt is being made.
To your point, you are storing the secret on your filesystem, and you should treat it accordingly. This is why folks generally advocate for the use of SSH Agents with password or other systems protecting your SSH key from being simply lifted. Even with requiring a password to unlock your key though, there's a pretty significant difference between key based and password based auth.
This might be similar to the workplace scenario that you have in mind, but some students are more bold in trying dodgy things with their class accounts, because they know they probably won't get in big trouble at an university.
The benefits is that it is probably much lighter weight than OpenSSH (which supports a lot of different things just because it is so general[2]) and can more easily integrate with their services, while also providing the benefit of not having to spin up a shell and deal with the potential security risks that contains.
And even if somehow a major flaw is found in OpenSSH, GitHub (at least their public servers) wouldn't be affected in this case since there's no shell to escape to.
[1]: I read it on HN somewhere that I don't remember now, however you can kinda confirm this yourself if you open up a raw TCP connection to github.com, where the connection string says
SSH-2.0-babeld-9102804c
According to an HN user[2], they were using libssh in 2015.
[2]: https://news.ycombinator.com/item?id=39978089
[3]: This isn't a value judgement on OpenSSH, I think it is downright amazing. However, GitHub has a much more narrow and specific use case, especially for an intentionally public SSH server.
Nobody is ever getting in, because they require ed25519 keys, but it is pounded nonstop all day long with brute force attempts. It wastes log space and IDS resources.
This is a case that could benefit from something like the new OpenSSH feature (which seems less hinky than fail2ban).
Another common case would be university students, so long as it's not applied to campus and local ISP IPs.
All the problems in this document are solved immediately.
Actually I might even want to let them into a "shell" that really screws with them, but that's far outside of ssh's scope.
According to to the article, you can exempt IPs from being blocked. So it won’t impact those coming from known IPs (statics, jump hosts, etc).
nobody cares about ips.
Is it possible to create some kind of reverse proxy for SSH which blocks password-based authentication, and furthermore only allows authentication by a known list of public keys?
The idea would be SSH to the reverse proxy, if you authenticate with an authorised public key (or certificate or whatever) it forwards your connection to the backend SSH server; all attempts to authenticate with a password are automatically rejected and never reach the backend.
In some ways what I'm describing here is a "bastion" or "jumphost", but in implementations of that idea I've seen, you SSH to the bastion/jumphost, get a shell, and then SSH again to the backend SSH – whereas I am talking about a proxy which automatically connects to the backend SSH using the same credentials once you have authenticated to it.
Furthermore, using a generic Linux box as a bastion/jumphost, you run the same risk that someone might create a weak password account–you can disable password authentication in the sshd config but what if someone turns it on? With this "intercepting proxy" idea, the proxy wouldn't even have any code to support password authentication, so you couldn't ever turn it on.
sshd_config requires root to modify, so you've got bigger problems than weak passwords at this point.
SSH is not HTTPS, a resource meant for the everyday consumer. If you know that you're behind a CGN, as a developer, an admin or a tool, you can solve this by using IPv6 or a VPN.
> Worst case, this will give bad actors the option to lock the original owner out of their own server
Which is kind of good? Should you access your own server if you are compromised and don't know it? Plus you get the benefit of noticing that you have a problem in your intranet.
I understand the POV that accessing it via CGN can lead to undesirable effects, but the benefit is worth it.
Then again, what benefit does it offer over fail2ban?
I would like to remind everyone that an internet facing SSH with a password is very unwise. I would argue you need to be able to articulate the justification for it, using keys is actually more convenient and significantly more secure.
Aside from initial boot, I cannot think of the last time I used a password for SSH instead of a key even on a LAN. Support for keys is universal and has been for most of my lifespan.
If I can be charitable ....
What the hell are you doing storing your SSH keys on-disk anyway ? :)
Put your keys on a Yubikey, take your keys with you.
I'm not arguing it isn't more secure. The point of this subthread is that SSH keys are not as easy to do ad-hoc as passwords, especially when moving workstations.
Extremely easy to recover from when the device you rely on to authenticate for everything gets lost or stolen too!
If I can't use TOTP with backup codes, I'm not using MFA.
ssh-keygen -t ed25519-sk
If I recall correctly, the SSH binaries provided by Apple don't have built-in support for signing keys, but if you install OpenSSH from Nix, MacPorts, etc., then you don't have to worry about this.
Another thing to be mindful of is that some programs have a very low timeout for waiting on SSH authentication, particularly git. SSH itself will wait quite a long time for user presence when using a signing key, whereas Git requires me to confirm presence within about 5 seconds or else operations fail with a timeout.
Carry your phone (many people already do this on a daily or near-daily basis in 2024) and use that in an emergency.
I've seen a few attempts to leverage certificates, or GPG; and keys nearly always are an 'easier' process with less burden to teach (which smart(er) people at times hate to do).
Of course OpenSSH also supports fido2 now but it's pretty new and many embedded servers don't support it. So I'm ignoring it for now. I need an openpgp setup for my password manager anyway.
I used pkcs11 before with openct and opensc (on OpenHSM PIV cards) and the problem I had with it was that I always needed to runtime-link a library to the SSH binary to make it work which was often causing problems on different platforms.
The nice thing about using PGP/GPG is that it can simulate an SSH agent so none of this is necessary, it will just communicate with the agent over a local socket.
Just curious: is it https://www.passwordstore.org/?
The one thing I miss a bit is that it doesn't do passkeys. But well.
[1]: https://github.com/android-password-store/Android-Password-S...
SSH Certificates are actually just a SSH Key attested by another SSH Key. There's no revocation system in place, nor anything more advanced then "I trust key x and so any keys signed by X I will trust"
At our org we simply distribute SSH public keys via Puppet. So if some leaves, switches teams (without access to our servers) or their key must be renewed, we simply update a line in a config file and call it a day.
That way we also have full control over what types of keys are supported and older, broken kex and signature algorithms are disabled.
I agree with the sentiment though, it is far less extensive than traditional X.509 certificate infrastructure.
You are correct though, you can keep a list and deploy it to all the nodes for revocation purposes.
It's unfortunate that there's no RevokedKeysCommand to support building something like OCSP.
And the SSH maintainers will refuse offers of X.509 support, with a justification.
Unless you regenerate them every day or have some kind of elaborate synchronisation process set up on the server side, a malicious ex-employee could abuse the old credentials post-termination.
This could be worked around leveraging TPMs, which would allow storing the keys themselves on hardware that can be confiscated, but standard user-based auth has a lot more (user-friendly) tooling and integration options.
I started using this when I got a new laptop and kept running into VMs and containers that I couldn't log into (I have password auth disabled). Same for some quick SSH sessions from my phone. Now, every time I need to log in from a new key/profile/device, I enroll one certificate (which is really just an id_ecdsa-cert.pub file next to id_ecdsa.pub) and instantly get access to all of my servers.
I also have a small VM with a long-lasting certificate that's configured to require username+password+TOTP, in case I ever lose access to all of my key files for some reason.
As I said above, keys are actually more convenient than passwords. Only reason people still use passwords is because they believe keys are difficult to use or manage.
And for those for whom it's an option, using U2F keys (like Yubikeys) is now easily doable with SSH.
So unless the attacker can hack the HSM inside your Yubikey, he's simply not getting your private SSH keys.
Yes there are good reasons. But it is quite a hassle to manage too.
Bullshit. You can have a terrible password and your system will still be nearly impossible to get into. Also, these attackers are usually looking for already exploited systems that have backdoor account/password combos, unless they are specifically attacking your organization.
Repeat after me: dictionary attack concerns have nothing to do with remote access authentication concerns.
Let's say my password is two common-use English words (100k-200k.) That's ten billion possibilities. Assume you hit on my password half-way through. That would be fifteen years of continuous, 24x7x365 testing at 10 password attempts per second....and then there's the small matter of you not knowing what my username is, or even whether you've got the right username or not, unless the ssh server has a timing-based detection attack.
The only argument for putting this functionality in the daemon itself is that by locating it in the daemon, it can offer advanced application-layer capabilities, such as failing auth attempts no matter what after the limit is tripped so that brute-forcing becomes more pointless - unless you get it right within the first few attempts, you could hit the right password and never know it. If they intend to implement features like that in the future, great - but if it's just going to do what fail2ban does, then...just run fail2ban.
Fail2ban has a higher overview of auth on the system, is completely disconnected from the ssh daemon in terms of monitoring and blocking, and the blocking it does happens at the kernel level in the networking stack instead of in userspace with much more overhead and in a proprietary system specific to SSH.
As a sysadmin, this is 'yet another place you have to look' to see why something isn't working.
Ok, let's try an example of a terrible password for the user "root": "password". Is that nearly impossible to get into? Or does that not qualify as a "terrible password" per your definition?
If your password is strong, it's not.
In a corporate setting, things are of course different.
Passwords are obsolete in 2024, and using them is very nearly universally bad.
Me neither. If your password has sufficient entropy, you don't need any of this.
> Malware can steal your credentials, it cannot steal mine
The only solution around this is a hardware key or MFA. I find the convenience of not needing anything with me to be superior to the low risk of malware. I understand your opinion may differ here.
The first claim is obviously nowhere near being true, and the second seems very subjective.
As the other user is saying, strong passwords with proper security have minimal risk. More than certs or keys yes, but they offer sufficient security and the balance with convenience is currently unbeatable.
Besides, even if someone gets access to your server they should be limited and unable to do any real damage anyway. Defense in depth and all that.
From the documentation, one of these parameters is in fact a group of 8 parameters. I guess the separator is space, so one could write:
PerSourcePenalties authfail:1m noauth:5m grace-exceeded:5m min:2m
Unfortunately, the default values are undocumented. So `PerSourcePenalties yes` (which will be the default value, according to the blog post) will apply some penalties. I did attempt to read the source code, but I'm reluctant to install a CVS client, two decades after dropping that versioning system.
PerSourcePenalties crash:90 authfail:5 noauth:1 grace-exceeded:20 max:600 min:15 max-sources:65536 overflow:permissive
[2] https://github.com/openbsd/src
[3] https://www.openssh.com/portable.html
[4] https://anongit.mindrot.org/openssh.git
[5] https://github.com/openssh/openssh-portable
[6] https://anongit.mindrot.org/openssh.git/tree/servconf.c?id=0...
> PerSourceNetBlockSize > Specifies the number of bits of source address that are grouped together for the purposes of applying PerSourceMaxStartups limits. Values for IPv4 and optionally IPv6 may be specified, separated by a colon. The default is 32:128, which means each address is considered individually.
Just like fail2ban, this seems like it can be equal parts helpful, a false sense of security, and a giant footgun.
For example, allowing n invalid login attempts per time interval and (v4) /24 is not a big problem for botnet-based brute force attacks, while it's very easy for somebody to get unintentionally locked out when connecting from behind a CG-NAT.
If you're doing that, I strongly suggest using ControlMaster to reuse the connections; it makes security tools like this less grumpy, but it's also a nice performance win.
I wonder how this SSH feature differs since it's implemented at the SSH level.
So long as the SSH and or PAM config requires more than a password (I use hardware keys), the main concern to me is log noise (making it hard to identify targeted security threats) and SSH DDoS. I know tarpits and alternative ports are another way of dealing with that, but when SSH is used for many things having to change the port is kind of annoying.
I think I'm probably just going to end up layering it like everyone else and stick everything behind a wireguard gateway, although that concept makes me slightly anxious about that single point of access failure.
Didn't the advice switch to using ipset a while back, precisely in the name of efficiency?
> I know the usual advice then is to reduce ban time and just not have permabanning, but that seems to kind of defeat the purpose.
The purpose is to make random password attempts even more impractical. With even fairly lax fail2ban rules, it'll take multiple lifetimes to find a password made up of just two common use english words.
However, that's not really their goal. I think that these SSH probes are mostly intended to find systems that have been compromised and have common backdoor passwords, though. ...and they use networks of zombie machines to do it.
That's where stuff like Crowdsec and IP ban lists come in, with the side benefit of your IP addresses becoming less 'visible'
> The purpose is to make random password attempts even more impractical. With even fairly lax fail2ban rules, it'll take multiple lifetimes to find a password made up of just two common use english words.
True, but the other reason to use such a measure is layered security. For instance I systematically disable all password only access, which kind of makes fail2ban seem a little pointless, but if there were to be some obscure bug in PAM, or SSH, or more likely a misconfiguration, then there is another layer that makes it more difficult.
This is useful in a defence-in-depth scenario, same as fail2ban. You might be able to defeat the odd hacker or researcher doing recon on your host, and sometimes that's good enough for you.
If you need botnet protection, you shop around for botnet protection providers, and you get a botnet protection solution. Easy as.
More importantly, though, is that it is difficult to debug why you can't login to a remote system, unless you've been diligent enough to setup remote logging and some kind of backdoor you can use in a pinch. I imagine many companies have some unimportant script running in the background that logs into a remote system over ssh, and the person who set it up left the company years ago. One password change/key rotation later, and suddenly 25% of employees cannot login to that remote system because the script got one of the office's 4 public IPv4 addresses blocked on the remote server.
It's very easy to say "you should manage your systems better, you should separate your networks better", and so on. But in my line of work (customer support), I only hear about the issue after people are already locked out. And I've been on many phone calls where users locked themselves out of their server that had fail2ban setup (ubuntu setup fail2ban by default in one of its releases).
This is the problem fail2ban (and now sshd) try to solve: I want a few people to log into my computer, so I open my computer to billions of other computers around the world and allow anyone to make a login attempt, and then I want to stop all the illegitimate attempts, after they were already able to access port 22.
It's simple Bayesian probability that any attempt to head off all those illegitimate accesses will routinely result in situations where legitimate users are blocked just due to random mistakes rather than malicious intent. Meanwhile, illegitimate attempts continue to come en masse thanks to botnets, allowing anyone with an ssh exploit the chance to try their luck against your server.
A more principled approach to security is to not roll out the welcome mat in the first place. Instead of opening up sshd to the world, allowing anyone to try, and then blocking them, instead don't open up sshd to the world in the first place.
1. If possible, only permit logins from known and (relatively) trusted networks, or at least networks where you have some recourse if someone on the same network tries to attack you.
2. If access is needed from an untrusted network, use wireguard or similar, so sshd only needs to trust the wireguard connection. Any attempt at illegitimate access needs to crack both wireguard and ssh.
With those one or two simple measures in place, have another look at your sshd auth logs and marvel at the silence of no one trying to attack you a million times per day, while also having confidence that you will never accidentally lock yourself out.
2. Doesn't wireguard then have the same challenge as SSH? Isn't that just pushing the problem around?
Another way to cut down on the log spam is by configuring sshd to listen on a nonstandard port.
Yeah, it's actually weird how frequently in those discussions people say some version of "just use vpn". I guess they really mean "just make someone else responsible".
It doesn't push the problem to wireguard. It requires wireguard to be broken as a pre-requisite for trying their hand at your sshd, and then they also need to break your sshd.
People who want this sort of thing and already have a single solution that handles multiple services have to complicate their setup in order to integrate this. They keep their existing solution for monitoring their web and mail server logs or whatever and then have this separate config to deal with for OpenSSH.
What if you don't want to refuse connections that exhibit "undesirable behavior" but do something else, like become a black hole to that IP address, and perhaps others in the IP range?
You want the flexibility to script arbitrary actions when arbitrary events are observed.
In my log monitoring system (home grown), the rules are sensitive to whether the account being targeted is the superuser or not.
Then you disable the behavior by turning it off in /etc/ssh/sshd_config.
Everytime when an authentication fails, you add the ip address to the xt_recent list in /proc and in iptables you just check via --hits and --seconds and then reject the connection attempt the next time.
Until then this has been a great differentiator for Bitvise SSH.
EDIT: To this feature’s credit, it’s not federated centrally, so a DDOS to nuke IP reputation would have its blast radius limited to the server(s) under attack.
On the jump host itself it makes sense though
it's a hard problem. wireguard has a pretty good attempt at it built into its handshaking protocol, but like all of these things, it's not perfect.
could maybe do something interesting with hashcash stamps for client identity assertion (with some kind of temporal validity window). so a client creates a hashcash stamped cookie that identifies itself for 30 minutes, and servers can do ban accounting based on said cookie.
https://weblog.masukomi.org/2018/03/25/zed-shaws-utu-saving-...
I am not a crypto guy, but my understanding is that users can downvote each other, and the more downvotes a user gets the harder the proof-of-work problem they had to solve before they post. If you received enough hate, your CPU would spike for a couple of minutes each time you tried to post, thus disincentivizing bad behavior.
I see on github the project is officially dead now:
Wireguard only responds to a complete useful key from incoming UDP (as I understand). Probe resistant.
I get the legacy argument here but it seems like almost two decades of "this tcp thing has some downsides for this job"?
Wonder if this is related to why Fail2Ban wasn’t in the pkg repos when I last tried to install it on OpenBSD?
There is only one thing on my wish list from the OpenBSD devs out there - that you’ll figure out Nvidia drivers.
It's the same with the OpenBSD operating system. There's so much extremely obscure, complex code that attempts to address the same problems we have been dealing with for 30+ years. What if we started removing code and reducing the attack surface instead of trying to patch over them, or we came up with an entirely new approach?
A good example of how code should be stripped down like this is WireGuard vs the old VPNs. WireGuard came along with fresh cryptography, took all of the bells, whistles, and knobs away and now provides an in-kernel VPN in a fraction of the LOC of IPsec or OpenVPN. As a result, it can be proven to be significantly more secure, and it's more performant too.
Have you used OpenBSD? You're telling them they should be doing something, that is already basically their mission statement.
Maybe OpenBSD itself is better at sticking to these principles than OpenSSH. I haven't used (experimented with) it for ~5 years but read about various updates every so often.
> Normal OpenSSH development produces a very small, secure, and easy to maintain version for the OpenBSD project. The OpenSSH Portability Team takes that pure version and adds portability code so that OpenSSH can run on many other operating systems.
Unless you actually run OpenBSD, what you think is "OpenSSH" is in fact "OpenSSH Portable Release". These are very different things.
There's plenty of flaws with any project, but OpenBSD is pretty well known for doing exactly the thing you're claiming they don't do.
Tinyssh is just one I already knew, I suppose you would find more via a proper search.
Edit: I realized after posting that I was equivocating on "complexity" a bit because you're talking about code complexity for openssh itself. I don't disagree with you that openssh itself would be less complex and more secure without password auth features, but I think it would have spillover effect that isn't a net positive when considering the whole picture.
I'd say this is a good thing, initial secret distribution is an unavoidable complexity and avoiding it leads to "admin/admin" logins which get hacked within seconds of internet access. There is plenty of tooling developed for this, even when setting up a VPS or flashing a Raspberry PI you can put a public key on the device to be active on first boot.
OpenBSD absolutely removes things: Bluetooth, Linux binary compatibility, and sudo, off the top of my head, with the sudo->doas replacement being exactly what you're asking for.
I've seen a number of reasonable criticisms of OpenBSD over the years. Not being minimalist enough is certainly a novel one.
They are know for doing exactly what you are suggesting.
Go ask @tedunangst. It was literally called "tedu'd" for ripping out old crusty code.