Fail2ban Sucks (2020)
j3s.sh
j3s.sh
I see this attitude frequently in online discussions around {security thing}, but it really doesn't match my understanding of reality. It's not about making any one part of the system perfectly secure, but instead about having all of the pieces of it contribute to a whole. Many real-life stories of caught/thwarted attacks include the threat actor successfully breaching/moving through multiple layers of a system but ultimately getting stuck on something relatively simple. Defense in depth is not just a catchphrase.
In evaluating whether something like this is genuinely useful, it can make sense to look at how difficult it is to operate, and how much it might reduce the probability of a successful attack. In my experience, fail2ban has been extremely easy to install and operate, and I think it brings at least some meaningful impact (particularly if applied to other things than SSH). At the very least it can cause someone making a focused attack to have work more slowly and take alternative approaches–"noisy" behavior that may itself trigger an alarm and get a human involved.
> do not change your standard ports. do not use fail2ban.
That used to be my attitude too. But reality intruded.
Example: I ran a SIP Server for a business. SIP Servers are very attractive targets because if you get past the security, you can make then call paid numbers and make real money. Knowing that, I used huge security tokens. No one got past them. But - it didn't stop them from trying. 100's of times per second at times. The CPU load created by processing the login attempts was so great the SIP server started dropping calls. As it happens, asking the kernel to drop packets from miscreant IP's takes far, far less CPU. Fail2ban to the rescue.
Example: Move on a few years, and now spammers costs have dropped so much it is worth their while to target IMAP, POP and SMTP servers. The story is the same: they never got through, but despite that their attempts increased to the point the server became overloaded. I can't listen on a different port. Fail2ban to the rescue.
Example: Move on to the last couple of years. I move my personal stuff to a VPS. To save costs it's very memory constrained. A problem soon arises. After a few weeks of trouble free operation it keeps slowing down, and sometimes becoming unresponsive. Turns out it's running out of memory; the OOM killer is running amok. That's odd, as there should be almost no load on this thing. Well, apart from ssh login attempts it seems. Turns out there is one of them every few seconds. They have no hope of succeeding of course as I only allow pubic key auth, but the load it creates is a problem. Fail2ban to the rescue.
I don't see anything wrong with the principle, which is you block these attacks at the cheapest point possible. That happens to be the point of entry, before you spend CPU cycles processing the packet (even in TCP). Fail2ban leverages that by using the kernel's networking stack. It's forced to glean the information on who to ban from log files, which is messy, error prone and fragile. I don't like that either, but I'll take what I can get.
And the author insists you shouldn't change ports any way. But seems to have no trouble just saying "use the security in your applications". Well... what sort of security is there in an IMAP server beyond "validate credentials"?
My other thought, reading the author's post, and having seen enough of this type of attitude, is that this type of person would be the one to say "just use... <external mail service>" or similar. "You can't ever do better than them!" Well... I have to suspect that techniques like fail2ban - blocking network access after X suspicious access attempts - get employed at the big service providers too. Possibly not fail2ban specifically, but the same ideas.
The solution we employed was a "dumb" fail2ban like script to ban any abusive IP address. It worked wonders and practically completely neutralized the attacks. We received zero complaints from any legitimate users of the service.
If I recall correctly, even after the attacks subsided (CVEs was patched etc.), we still kept a toned down version of the script in place.
> at best, fail2ban: does nothing
This is just plain wrong. Even if you're only using key based auth and there's no way an attacker could have your keys, fail2ban will still block the abusive traffic.
I have no issues with blocking abusive traffic instead of letting the attackers beat their fists against my armored doors.
> at worst, fail2ban: causes you massive inconvinience or total lockout
The authors main criticism seem to be that you might lock yourself out of your own server one day. This hardly seems fair because the author is saying "just setup SSH correctly and you won't have any issues" while also saying "if you don't setup fail2ban correctly, you'll have a whole bunch of issues!" OK, so shouldn't we just setup fail2ban correctly as well?
Having said all that, my recommendation for people who are going to use an indiscriminate fail2ban setup on SSH access would be to have some out-of-band access in case you do happen to ban yourself, even if it's less convenient to use than SSH.
I wonder if the author has ever used KVM, iDrac, IPMI, etc., or the log-in consoles on cloud platforms.
And if you don't have those then 1) woe be to you, and 2) pay the 150/hr remote hands fees to get someone to console in for you.
(I'm not advocating for a change; I'm just unfamiliar with the arguments for both sides.)
The examples where there to demonstrate the conclusion in the final paragraph matters in real life. The para could stand on it's own - it's just a statement of the obvious. But like all obvious things, it's only obvious once you see it. Like the author of the article, I didn't see it for years (decades?).
Were you thinking that I was arguing that e.g. you shouldn't be able to use a pseudonym to post comments?
Sorry if that's how my comment reads. The thing I was wondering about was if/why we should allow spammers / crackers to retain their anonymity when attacking computers in the way you described. E.g., 100 attacks / second on some port, with no accountability for the persons who originated / propagated those packets.
I'm curious about the pros/cons of law-enforcement agencies having the ability (and mandate) to hunt down the sources of such traffic.
I realize that doing so would open a huge can of works, so I'm 10000% not advocating for it. I'd just like to understand the steel-man arguments for and against.
fail2ban also works with services other than SSH, and one of my favorite tricks is to point it at nginx logs, add a `robots.txt` with a honeypot entry and have fail2ban block any IP that hits that or any of a dozen common vulnerability endpoints that I know aren't on the server (like `wp-admin.php`). Keeps the web logs cleaner and more useful.
More seriously, I don’t see why anyone would want to disable fail2ban (outside of purposefully being against the grain) because it seems like “free” protection. I’m sure there’s some situations where it may be beneficial to, but those seem like extreme edge cases.
---
I dont need to remember non standard ports if i just use my .ssh/config.
Host myhost.tld
Hostname 127.0.0.1
Port 333
User MyUser
IdentityFile ~/.ssh/myhost.tldI know security through obscurity is not an answer, but judging by the reduction in port scanning i've seen after moving as many standard ports as possible to new addresses above 20000 I have to believe its a reasonable first step. How many script kiddies are scanning all 65500 ports for each IP address?
The next trick I think of implementing is port knocking. Should drop log noise to zero unless someone starts targeting me specifically. In which case, my goose is already cooked.
If you're at that point, I would suggest putting it behind wireguard.
Consider this: an attacker (somehow) managed to get user access to your server. They can now dos the service until it crashes and then start their own service listening on that same port, maybe impersonating your service. Maybe they can use that to grab sensitive information or do something else.
Worse case Google the commands and run them
And even only in the OpenSSH universe, I find it quite annoying having to remember whether the `-p` can go after the hostname or has to appear before, whether SCP uses `-p` or `-P`, whether `ssh-copy-id` supports one or the other etc.
There are some protocols I wouldn't necessarily run on their default port and publicly accessible, but SSH is really not one of them, also given that sshd has been specifically hardened for that adversarial use case.
scp myfile.txt myhost.tld:~/fooBut then again, SSH is often not considered essential and you need a VPN or other tunnel on TCP/443 anyway...
Normally I might put a title like this down to opinion but here, the author suffers not knowing enough about the subject to justify having an opinion, let alone publish one.
Fail2ban is a stupidly easy way to block lazy hacking scripts. It's easy to extend to handle simple honeypot services. It's no replacement for real security but it makes focussing easier.
In fail2ban's defense, it can be used for a lot more than just blocking IP addresses based on SSH failures.
At one point in the past I adapted an existing fail2ban setup to ban IP addresses that were running malicious vulnerability scans against a large legacy PHP webapp, using some dumb heuristics that worked and were easy to put in place.
Would it have stopped sufficiently motivated expert attackers? Of course not.
Did it reduce the load on the legacy PHP app and improve the signal to noise ratio in our webserver logs, while reducing the chance a brute-force attacker would find some hole lurking in a long-forgotten corner of the app's unfortunately-large attack surface?
Yup.
So, I'd argue it can be useful, even if naive SSH blocking may not offer much improvement.
Brute force protection is completely unnecessary for openssh when you've disabled password authentication because it's impossible[really expensive] to brute force. Until you are impacted by an authentication vulnerability which still requires a valid user, then you are brute forcing a user away from being totally compromised. All of a sudden doing all the "unnecessary" things, like brute force protection, disabling root logins and not using default accounts make the difference between you being at the front or back of the line to get compromised in the race to get patched.
If you are one of those who believes a vulnerability like that is extremely unlikely rather than inevitable, then consider this. An ssh key gets leaked, the system it has access to is known / discoverable, the username is not. Brute force protection is now the main difference between being highly at risk and still reasonable safe. It's a contrived scenario, but hardly an outrageous one.
However, the moment the industry went to SSH Keys en masse, and got away from passwords, fail2ban stop serving any real purpose other than to make people feel like they had done something to improve security. Which it didn’t really, especially if you enforced key usage only for logins.
Literally, the only argument you can make is that fail2ban might reduce the number of log entries.
That assumes people only use it for sshd, no? Which isn't the case [anecdatum] for me, at least - I use it for ssh, http (several servers), imap, smtp, mqtt, etc. Make a request for phpMyAdmin pages on my server? 10 day timeout for you, sir. IMAPS connection with incorrect username? 10 days in the bin. etc.
Yeah, it reduces log entries but it also reduces unwanted resource consumption and keeps my servers working instead of chugging under non-useful load.
So it's not just for password based SSH.
The only time I almost used Fail2Ban was with a personal "guaranteed to work from wherever and whatever I'm on" jumphost which ran SSH on 22 and 443. 443 because, like the post says, outbound filter compatibility (Fun note: to listen on multiple ports simultaneously you can literally just include multiple "Port xyz" lines in your sshd_config). Ultimately I decided for this use case it made more sense to just let bad attempts to fail silently by changing the logging config. If I was ever in a situation I felt I needed the failed attempts logged I'm not sure I could simultaneously justify username+password login on a public host.
Let's be clear here - fail2ban will not save you if you have a poorly configured SSH server or widely open Nginx/Postfix services, but it will offload traffic blocking to iptables (or whatever firewall you are using). This way, the mentioned services will not have to waste resources just to cancel some connections.
Will it save it against large DDoS? Surely not, but it will stop mild DDoS without problems, which it is designed for. We can argue that iptables/nftables can do a fail2ban job with rate limiting and other techniques many people have never heard of, but if you are at this level, you already know your craft, and I'm pretty sure you considered, tried or used fail2ban before.
If you're relying on privileged ports, you've already lost the battle.
* your SSH client will check the host certificate. There's a reason it does that prompt for new hosts!
* a userland process can no more MITM that than any random middlebox can.
Non-standard ports.
They work wonders against lots and lots of traffic from Chinese bots which are too stupid to see that password logins are disabled but there's a fucking lot of them, they just won't stop knocking, and it does take cycles to fend them off in any way. You stop listening on port 22, and 99% of that shit traffic just goes away.
I know, it's not by the book, and that well ackchyually I should yak yak this and yada yada that. But it's so fucking cheap and simple to do, and it goes brrr, so I just don't care.
> changing your SSH port to non-root port (>1024) means that if the SSH daemon dies, any userland process can take its' place. this is bad.
Man, if you have sshd that likes to die a lot, you've got bigger problems already. If you have userland processes that can take your sshd port over maliciously, you're already fucked and should really reprioritize your worries.
> changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working.
~/.ssh/config goes brrrrrr
> you will very likely waste hours of your life.
Only if I need to dictate the contents of that file to my iPhone, otherwise it's seconds.
> you will have to google "rsync ssh non standard port" every time you want to use rsync
rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr. Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin?
> you will have to remember scp flags. this is also bad. probably worse.
scp from OpenSSH honours OpenSSH configs, so it goes brrrr.
> you will have to tell the port to anyone else who tries legitimately accessing your system.
"Here's a fragment that you copy and paste into your ~/.ssh/config. Remember to do a chmod 0600 on it if the file is not there already."
My god, is it me getting old, but when I was young it was customary to know the basics of your tools.
It doesn't need to happen maliciously. The result is the same: You might be locked out permanently.
> if you have sshd that likes to die a lot, you've got bigger problems already
It doesn't have to die, it can also just happen during startup.
Let's say you initially configure port 2222, restart sshd, everything works well, so you remove port 22, restart one more time, and stop thinking about it.
Then, on your next reboot, it turns out that port 2222 is actually the preferred port of some other automatically starting service that just happened to not be running during your experiment, but which is now racing sshd for port 2222: That's bad.
> rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr.
And just as many applications including an SSH client won't pick up the config from .ssh/config (for example because they can't read that directory because they're sandboxed).
> Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin?
SSH is used by all kinds of people, not all of them condescending sysadmins.
> Then, on your next reboot, it turns out that port 2222 is actually the preferred port of some other automatically starting service that just happened to not be running during your experiment, but which is now racing sshd for port 2222: That's bad.
Isn't it sysadmin 101 "How not to get locked out"? You should know already what else might be running on the box, or learn the hard way.
> And just as many applications including an SSH client won't pick up the config from .ssh/config (for example because they can't read that directory because they're sandboxed).
Put the relevant config where the sandbox can see it.
> SSH is used by all kinds of people, not all of them condescending sysadmins.
SSH should be used by people who at least know what a file is and how to protect their private key, I think that putting in a few lines of configuration (or having a provisioning tool do it for them) is a non-issue. Except if you are willing to die on that hill and think up artificial obstacles.
This one.
One amusing thing what these apologists completely ignore (because they don't know shit?): just fucking DNAT a high port to localhost:22 and drop externalip:22. Bam, works.
yes, it sucks to make efforts to accommodate your worst software, but sometimes it's easier to throw the cycles away than to dick around with a custom iptables entry to fix someone elses poor software design choices.
Good part of being me is not having to deal with such tools. If I see something like this, I consider finding an alternative immediately because the hardcoded port will be the least of its problems. If it assumes and insists on a port, there's a good chance it doesn't even piggyback on an ssh command or libssh and reimplements lots of ssh itself. What ever could go wrong with this approach?
If it's a tool to which source is available, it can be patched up. It can be used on the right side of the LAN where you can use well-known ports all you want. Embrace and use VPNs, they are good! I'm somehow convinced that if the ground rules are laid out up front in no ambiguous terms, the workaround will be found on the side of the badly-written software.
However, scanners have changed drastically during the last 5-10 years. I remember when you saw close to none login attempts when you used a port between 15000 and 20000. Nowdays there are 100s every day. I guess on 22 it's much worse, but I have not used 22 just for the sake of comparing.
For fail2ban it's the same: I know it's not a great solution (lightweight code base, better security gains and lower risk to affect legitimate users would be great), but there is no convincing argument and no better alternative in the submission. Good security for the whole installation is not an alternative, it's a precondition for connecting anything to the internet anyway.
At times, for bastion hosts, I use crowdsec instead.
I could not capture why the author was so bitter. I guess I would be the same if I was supporting some product, and its users were getting problems because of fail2ban - and I had to support it for them.
I agree that fail2ban is useless once you've disabled password login, which you should have done.
But is anyone really getting locked out of their own systems by fail2ban unless they're doing something pretty weird? I thought it gave you about 5 password attempts, and then blocked you for 10 minutes - doesn't seem that draconian to me.
[1] https://man.freebsd.org/cgi/man.cgi?query=blacklistd&sektion...
And yes, DoS is a very good example, if your application writes so much log's (during a attack) and fail2ban has to analyze them during that, then machine could very well DoS itself, the blacklistd approach is much simpler and straightforward BUT (and that's a big one) the applications has to support it.
A few years ago I had to migrate an ancient VM running OpenSSH (for SFTP only) to our k8s cluster. 99% of the users had passwords instead of keys. Easy enough to run sshd in a container with a mounted volume, but I wanted something to recognize and block repeated bad login attempts (the old VM had no such protections).
sshd doesn't do it natively, fail2ban wasn't a good option since you generally don't want a pod messing with a k8 node's iptables. What I ended up with was a Ruby script with 3 threads: One to monitor the auth log for failed logins, a second to lock accounts with repeated failures during a given window, and a third to unlock them after some period (and in the darkness bind them). Definitely some caveats in there, but it was an improvement over the status quo.
i do hate the idea of ip whitelisting; IP's should be more dynamic and less able to fingerprint an individual - but that is opening another can of wurms. Generally never trust ANY ip; dont use whitelists, and the blacklists should never be "forever". if your app is secured by a keypair like openssh then thats the route to take.
honestly though fail2ban is great in that it does silence some noise in the logs - there will still be an unbareable amount - but at least some less; but still regular log maint is required. Also the groups Ive seen never recommend f2b for use to fight against a DoS attack; not even for "cleaning up logs" - i dont see where that point was created other than from this page
If ssh is only allowed on the management wireguard network, not much can beat that other than a host compromise which, additional checks to verify that a host can be trusted at that moment could potentially help stop unauthorized access.
It is simple and elegant and we're coming up on ~20 years of deployment in various places with zero hangs or crashes or unexpected interactions.
While nobody should rely solely on port knocking, it is a wonderful layer added to a defense-in-depth strategy.
Which is to say: lock down everything the way you always do, then put the port behind a knock.
The knock utility (and associated knockd daemon) are simple and resilient - don't make it more complex.
Instead, add a simple line to your .login (or whatever) that sends you an SMS every time a successful (three packet) knock occurs.
You're the only one that should perform that ... so it would be very interesting if a successful knock occurred without your having knocked ...
https://www.kylheku.com/cgit/txrban/
It's way simpler. There is a tiny core to it, and a couple of modules specific to Exim, Apache, SSH. These just run as daemons on the machine.
I tried using fail2ban at that time, but it wasn't at all what I expected. It was way overcomplicated for what it achieves, and poorly documented. Just getting it running at all was fiddly, never mind the custom behaviors I had in mind.
I should review the Apache module; some of it is quite out of date, due to the changing landscape. Also, that webserver it is used with is now behind a reverse proxy, which only receives connections for specific applications.
Home paged here: https://nongnu.org/txr
it's easy to setup and can stop multi stage attacks that generate quirky log output.
EDIT: fail2ban vulnerabilities? what is this guy talking about?
There have been several: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=fail2ban. Mostly DoS. At least one RCE, admittedly with a non-default fail2ban config and a somewhat unlikely attack chain: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3274...
you had to manipulate answers from a whois server.
Sure, it increases attack surface (like any additional piece of code) but argueing to better let an IP hammer your mailserver with infinite stuffed credentials just to avoid possible bugs, is questionable at least.
fail2ban's CVE track record is quite good in comparision, if you take that as metric. It might be even more secure than using multiple different rate-limiting implementations that were written in C.
For me, it is less of a hassle than looking at logs flooded with failed connection attempts.
Also, I sometimes want to know about failed attempts, mostly for legitimate attempts that failed and shouldn't have. As in: "why couldn't bob connect the the server?", and failed attempts would just show that he used the wrong password, or at least, that it is a password related problem.
[0] ie Morris worm
In any case your threat model for container workflow is already 100x more complicated than some one-server blog with an open SSH port, I would give up on fail2ban immediately
Just two days ago I found out the sensible defaults doesn't include automagically enabling even the basic jails. The machine run for a year (or more) with enabled fail2ban which... did nothing. Not a Debian machine and it's entirely on me, but fail2ban setup is not "just apt install". You need to be at least somewhat familiar with it's lingo and the minimals on working with it.
But at some point I got too many log entries of failed ssh access attempts. So I looked into fail2ban, but found the system too involved, and I don't like python for this kind of thing to start with. Then I started looking for alternatives, because I figured you only need something very simple, that checks certain log files and then instructs the kernel through netfilter to drop traffic from all infracting IPs. I like single-binary applications, especially for things installed outside the normal package manager. Found a skeleton of a golang app that claimed to work (it didn't), but I managed to rework it, and it serves my use cases: https://github.com/pepa65/fail2drop
The real advice for setting up new servers IMO is, SSH pubkey auth is perfectly adequate security. Set it up normally on the default port logging in as root, disable password login, and don't worry about the noisy failed bot logins. This is a lot more convenient for automated setup and maintenance and has no practical loss of security.
Worse? How?
Hrm... If I don't use commands for a while, yes, I may have to remember them. If that's the tradeoff for one extra layer providing even a modicum of security... why is this a bad thing?
We have things like bash scripts... where you can just store the common flags you use for common things.
I wonder how often the author port tunnels over ssh? I usually have to go search for the syntax for ... 30 seconds. Then I have it. Or it might be in my history already.
"You might have to remember something" seems about the weakest argument against changing ssh ports I've ever read.
Problem is DNS query can come from anywhere so latching to the top-level domain (TLD) name became a must.
Fail2ban, to the rescue.
https://egbert.net/blog/articles/troubleshooting-regex-in-fa...
I often use a combination of:
-blacklistd and fail2ban (for applications who don't support blacklistd)
-pfbadhost (nearly always)
-CrowdSec (just sometimes, but atm pretty happy with the free service)
I use those to have clean logs, block spammers at ip level, and "marketing" crawlers...well and other stuff.
And no, ssh is never on 22, i like clean logs to better find the real "interesting" stuff.
Internal users' machines have these tunnels transparently loaded and running, configured by our internal GitOps-driven config management (SaltStack).
Has been pretty comfy so far.
It may not seem like much, but I spent a year or two adminning a >1 million reqs / day cluster, and it does add up. Auto-banning this type of obvious offender (with fail2ban, as it happens) did result in a noticeable improvement for us.
Fail2Ban can prevent password spraying from working, since an eventual good password will still be blocked if done within the fail2ban block period. And those block periods aren't publicized.
And given appropriate configuration, is another layer of security a potential threat has to deal with.
I have not the best memory but after using it several times I managed to remember "rsync -e 'ssh -p 223' ..." and "scp -P 223". It wasn't that hard really.
Fail2ban has never seemed to be the sophisticated or final solution, just a bare minimum.
Inconvenience to users is definitely a form of the security vs. Convenience debate.
But that's not a reflection on the fail2ban maintainers, the engineering work that's gone into building it, its usability, bugs or indeed anything else connected to its implementation.
It sucks simply because it's solving the problem in the wrong way. There is no sensible security onion, or defence in depth model for open ports in front of private systems.
If the technical solution for keeping private, or limited access systems actually and effectively private on the public Internet requires open ports, it's the wrong solution.
Unless we're running services designed for public and anonymous access, exposing technologies like VPN servers, SSH or any other software for private consumption to the Internet is a mistake.
When we do this we're putting out invitations for abuse and pulling the burden of security responsibility onto ourselves. I'd argue however, that unless it's a full time concern for us, we're close to the project or maintainers, and able to contribute security fixes to source code, we're probably ill equipped to directly handle that responsibility and remain dependent on engaging with the surrounding security eco-system to remain secure.
I'm not knocking those security eco-systems, I'm just pointing out that there's a time lag inherent on depending on best-efforts from third parties which creates windows of vulnerability, and open ports on infrastructure ALWAYS puts us on the back foot. From getting patches applied in a timely fashion to the time and convenience cost of updating ACLs each time IPs change. Even if we get the maintenance tasks right 100% of the time, we're still open to the risks of something outside of our control happening, like zero day being sprayed across open ports.
The U.S. Government correctly issued an executive order mandating Federal Agencies move to adopt Zero Trust principles just six days after the Colonial Pipeline ransomware attack in 2021. Despite the marketing hype which has followed, if we take nothing else away from this motion, it should be that opening ports in our networks to access private systems using the public Internet is over.
There are at least 90 projects and businesses today dedicated to building modern private access technologies which allow secure remote connections and access to private networks without opening firewall ports. Many are commercial options serving businesses, but there are also lots of compelling open source offerings for non-commercial use too.
There's a directory of vendors and technologies here https://zerotrustnetworkaccess.info/ which attempts to dispense with some the Zero Trust marketing BS and instead focus on technical discourse, architecture and approach which some might find helpful.
Disclosure; founder of one of the businesses (enclave.io) with a commercial interest in this space.
The problem apparently is that Apple Mail is pretty aggressive at retrying if a login fails, and so if it tries to connect before you have finished changing the password in the client it can try enough times in quick succession to trigger Fail2Ban.
There are at least 3 places you can change your password(s) for Apple Mail.
• Invoke "Mail/Settings..." in Mail to open the settings dialog. Select the "Accounts" tab. Then for the account you want to change go to the "Server Settings" pane. That has password fields for your incoming mail server and your outgoing mail server.
• Invoke "/System Settings...", then go to the "Internet Accounts" settings. (You can also get there from "Mail/Account..." in Mail). Select you mail account, click the "Detail..." button, and you'll get a dialog that has a password field.
Note it only has one password field. I'm not sure what you are supposed to do if you have different passwords for you incoming and outgoing mail servers.
• Open "Keychain Access", and search for your incoming mail server. Look for an entry whose kind is "application password" named com.apple.account.IMAP.password. (I assume that IMAP would be POP3 if you use a POP3 server). Open it, verify that the Account field is you@that_server. Then you can click "Show password" to reveal the current password. Change that to the new password and save.
Do the same for your outgoing mail server, looking for com.apple.account.SMTP.password.
My current procedure is to first disable my work mail account in "Mail/Settings...", then change the password on the server, then update the client side passwords via Keychain access, and then via "Mail/Settings...", then enable my work mail account. That seems to usually work, although once it didn't until I also changed it in "Internet Accounts" in the system settings. I also had a time where I just could not get the new password to take until I rebooted.
Apple Calendar is also a pain with password changes, but we don't have Fail2Ban on our calendar server (it is not on a public facing network so there would be no point) so it is not as annoying as Mail.
For Calendar what I've found to mostly work is:
1. In "Calendar/Settings..." in the settings for my work account in the Accounts tab turn off all delegates. Then disable my work account. The quit Calendar.
On MacOS prior to 13 find and kill the CalendarAgent and CalNCService processes. On 13 they went away.
2. Change the password on the server.
3. Change the password in "Keychain Access". I find the entry by searching for name@calendar_server. The right entry is the "application password" named "com.apple.account.CalDAV.password".
4. Then change the password in "Internet Accounts". As with Mail, Calendar provides convenient access to this via "Calendar/Accounts...".
5. Open Calendar, reenable the account, and turn back on delegates.
All of the above comes from trial and error and I have no idea how much of it is actually necessary. I haven't had the patience to do the time consuming experimentation it would take to figure out the actual simplest way to change these passwords reliably and without hitting the server with the old password during the changing.