Fail2Ban | Hacker News Reader

Fail2Ban

github.com

160 pointsredbell128 comments

I wrote about using fail2ban with postfix[0] almost 15 years ago for blocking IPs that were sending to unknown email addresses on my server. It was a godsend for blocking tons of spam back then.

[0]: https://www.dp.cx/blog/postfix---fail2ban---win.html

fail2ban is a real pet peeve of mine because anyone security conscious enough to deploy this will have likely already mitigated any actual security risks this could help with either by using a strong password or public key authentication.

That leaves noise in the logs - which sure, it's nice to reduce, but using an alternative port can help here.

I may sound like a spoilsport - but the fact that there have been a number of security vulnerabilities (https://www.cvedetails.com/vulnerability-list/vendor_id-5567...) in this project, make it worse than security theatre, it actually increases risk whilst not at all reducing it.

Yes. At this point, fail2ban has become almost a shibboleth for people following security checklists as opposed to reasoning about a coherent threat model. This is a perennial topic on HN, and almost always devolves to some appeal to grooming logs, because of all the authentication errors fail2ban is presumably preventing.

Don't use fail2ban. (Don't use passwords, either!)

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

I am one of the people to whom you refer. I read about fail2ban in a "Linux Server Bible" e-book around 2010 and have used it on all of my servers since, even though I am careful with my keys and use password-less login.

Does fail2ban have authorization to write firewall rules? That's a high-impact vector of attack, should fail2ban have a vulnerability. Also, does fail2ban store credentials that provide that authorization?

Yes, no.

It runs as root.

I use fail2ban because I take break in attempts personally, especially when it's some script trying default logins one after another. It's insulting.

> It's insulting

Brute force / credential stuffing attacks against ssh are the mosquitos of the Internet.

Ubiquitous, annoying, and persistent. But nothing personal.

I also take mosquitoes personally, so maybe it's a larger character flaw on my part.

You have that exactly backwards: if someone is hitting you with a password bruteforce from a single IP address (which is the only threat that fail2ban mitigates) then it is assuredly nothing personal at all.

A personal insult, if you are ever unfortunate enough to receive one, will be much more stealthy and neither fail2ban nor any other magical rock will protect you against it.

You don't see any tigers around, do you?

Usually they use a big pool of IP addresses, but that doesn't make fail2ban completely useless since they do reuse IPs.

If anything, you're doing them a (miniscule) favor by keeping them from wasting more resources on failed login attempts. If you really hated them, you'd set up a honeypot.

More fun: setup a fail2ban actionban script that instead of banning the IP, shapes the traffic coming from it to have abysmal bandwidth so requests/responses takes really long time, so they'll have to timeout instead of getting failures.

This is known as tarpitting, and apparently iptables can do it: https://en.wikipedia.org/wiki/Tarpit_%28networking%29

This is hilarious

Neat, didn't know that! Think before I've used Traffic Control (tc) for it, but iptables would be simpler.

Available in `xtables-addons` it seems. After install:

    iptables -A INPUT -p tcp -s $SOURCE_IP -j TARPIT # add IP to tarpit
    iptables -D INPUT -p tcp -s $SOURCE_IP -j TARPIT # remove IP from tarpit

Restrict your SSH login to a single user, then su to your admin

I know how to lock machines down. That's not why I bugs me.

Apologies, I guess I just wanted my two cents in and didn't see anyone writing it

No, it's solid advice. Thanks for looking out for others.

Grooming logs from attempts seems like shibboleth on its own that is indicating junior level or “security enthusiast”.

Anyone who manages servers professionally does not read logs anymore and does not care about obvious things like people brute-forcing.

Reading ssh logs on your single VPS is security LARPING. Discussing faill2ban as well :)

As that gaps of inequality of our winner takes all societies ever widen, anyone just trying to do something for themselves and not for global scale SV company is just meaninglessly role playing.

Not really. Only that reading logs by human or even grepping "manually" is super inefficient where you can make a script that will for instance send you a notification when someone actually logs into your VPS.

In world where "login attempts" are basically all day reality reading logs is meaningless. In 2023 no one should be reading logs you should have alerts on events. In 1995 or something if someone was trying to brute-force your user password that was security event to look at and block IP. In 2023 someone brute-forcing is not an event, it is either wrong configuration like not using ssh authentication or not using tools that filter logs automatically and make alerts when something is actually going on.

While I agree fail2ban is a wrong tool to prevent password brute force - better authentication mechanisms should be used instead - it has its uses. For example, it can be used to automatically ban (or alert about) dumb http scanners like gobuster. I am not saying, a determined attacker cannot bypass it, but if it saves me some hassle and raises the bar for them, why should not I do it?

More general, some attacker actions, especially during recon, rely on making many attempts to connect, fetch an URL, resolve FQDN, etc., these could be detected and automatically responded to, making attacker’s job harder and providing extra visibility to defenders.

You shouldn’t use it because fail2ban itself can (and has been) attacked. It doesn’t make the attackers job meaningfully harder but does add complexity to your systems, that complexity is weakness.

I looked at fail2ban exploits and they are either LPE due to file permissions or command injection in other tools like mailutils.

Citation needed for the claim “has been attacked” if you refer to real attacks in the wild.

One noted upthread https://news.ycombinator.com/item?id=37796588

Yeah, that’s command injection in mailutils I mentioned, not in fail2ban itself. Did you see how it’s supposed to be exploited? Did you see a real-life exploitation?

While it’s a nice trick, it’s simply not relevant. And the vulnerability before that seems to be 10 years old. I’d say it’s a decent track record.

Doesn’t it help to mitigate DoS type attacks by reducing the amount of CPU that a bad actor can burn?

You can do it with ufw limit too

You can also literally have anything pipe rules into it. Want WordPress auth to result in fail2ban-enforced bans? You can do that. Want cheap rate limiting? You can do that too

This thread seems pretty full of people dismissing the project based on the idea that it only protects against ssh credential stuffing, and ignoring the other 99.5% of what F2B does.

If someone is performing a denial of service attack from one I.P. address then this will help.

To tptacek's point, you've got to ask yourself is a denial of service attack in your threat model?

The reality is most folk set up fail2ban after seeing auth failures in their logs, not service degradation.

If you're considering a denial of service attack in your threat model, then I'd probably also consider a DDoS attack and there are likely more effective solutions here (a firewall or CDN).

And don't forget you're using some of those precious CPU cycles to parse the auth logs, with python no less :-)

>And don't forget you're using some of those precious CPU cycles to parse the auth logs, with python no less :-)

You can ship the log somewhere else, do the fail2ban there and perform the block action in another place up the stack.

f2b can also do an (r)whois lookup and ban netblocks.

Yup. I see many resources for self-hosting recommend fail2ban for e.g. SSH. But I always disable password-based SSH logins on all of my computers. The one niche use case I can see for fail2ban is possibly reducing the amount of hits to /wp-login.php and /cgi-bin in your web server (or reverse proxy's) access logs.

"Don't use fail2ban because you don't need it if you do XYZ"

I'm not so sure that's a good reason to be honest. And if you're worried about CVE's, well, you'll be using handwritten, hand delivered notes before long. Keep your systems patched, keep them tidy, none of this is likely to affect you, fail2ban or not.

To put it another way - there is no security risk that fail2ban helps with that can't be resolved in another, better, more robust and less risky way.

But it also helps in reducing the load on your servers when, e.g., instead of 300+ login attempts per minute on your mail ports, you get 20 because the IP gets banned for a day after 2 failures. Or, instead of nginx spending 90% of its time sending out 404s for the various PHP and MySQL holes I do not have installed, it can spend 10% of its time instead.

Particularly on my small server, fail2ban is the difference between "usable" and "on the edge of falling over".

Parent says there are more robust solutions to these and there are. Rate limiting is one that has been in use forever for example.

Yes, I'm rate-limiting by using fail2ban to drop traffic that I don't want.

If you’re a hobbyist sysadmin setting up a personal VPS then the security risk is your own competence in correctly configuring things the better more robust less risky way, but you can’t replace yourself with a more competent sysadmin in this scenario, so fail2ban helps to Swiss cheese model this edge case.

Saying anyone who makes mistakes is just incompetent is really just a “no true Scotsman” argument.

Everyone makes mistakes. That’s the whole point of the Swiss cheese model and of layers of security in general.

Excuse me, if fail2ban is frowned upon, what is the alternative to block crawlers that try to find wordpress or php endpoints on my website, two software that I don't have installed?

The idea is you don't have to block those since there is no attack surface.

I look at the imap login attempts on my server sometimes. The passwords they try are usually pathetic. Nothing close to the 15+ character actual passwords we have in use.

So the idea is I shouldn't need an alarm system in my house because all my valuables are kept at a safe that can't be opened by anyone but me?

I disagree with this, 404 queries still use resources and someone trying URLs in a matter of seconds should be blocked nonetheless.

> […] that can't be resolved in another, better, more robust and less risky way.

Only if you can get business/users/management buy-in or approval for implementing those ways and changing workflows.

> That leaves noise in the logs - which sure, it's nice to reduce, but using an alternative port can help here.

No, it cannot. As a sysadmin I do not want to get into user training about telling people about alternative ports and tweaking their CLI habits and any scripts that they have.

If you want to further cut down on the log noise get an IPv6 address (and drop IPv4)—good luck to anyone trying to scan a /64 for open ports.

You can scan ipv6 because the addresses aren't arbitrary. Blocks have to be purchased and then ranges within routed.

Individuals usually get a /64. Scanning a truly random address in that range is not feasible.

You can try, but a lot of ISPs assign a big subnet to each user. Mine for example assigns a /48 to each home user fiber connection.

Even if I make no effort at all to hide things and just select xxxx:xxxx:xxxx:1:: as the subnet (leaving a factor 65535 options on the table) the devices behind it will randomize the next 64 bits meaning you'll have to scan 18 quintillion (1.8e19) addresses to find one.

I read that Shodan was running NTP servers to figure out active IPv6 addresses :)

I can cofirm this, I swapped one of my cloud VMs to ipv6 only ssh and after 11 months I never seen a single ip besides mine attempt to login. This was using the default port 22.

My lame provider (comcast business) wants $20/mo for ipv6.

I'm rather convinced that people reaching for fail2band actually want rate-limiting.

Often, they do, however configuring it for different applications may be a bigger effort than doing so via fail2ban with minimal log parser tweaking.

This is true but on an active server legitimate users getting blocked far exceeds the convince of having cleaner logs in my experience.

(Blocks always have to do with saved passwords being used from a nonwhitelisted IP)

One nice thing I'll say about fail2ban is that it can fire off reports with decent logs to the networks responsible alerting them to compromised systems and bad actors.

> That leaves noise in the logs - which sure, it's nice to reduce, but using an alternative port can help here.

Shifting services to alternate port numbers will stop very stupid scanners but it does not stop the worst offenders IME. Basically it just means you'll only get the really obnoxious sources that try everything ignoring responses.

> I may sound like a spoilsport - but the fact that there have been a number of security vulnerabilities (https://www.cvedetails.com/vulnerability-list/vendor_id-5567...) in this project, make it worse than security theatre, it actually increases risk whilst not at all reducing it.

Given the age of the project and that there's been a whopping NINE vulnerabilities found in its lifetime, this is a great take. By this same logic you better disable OpenSSH everywhere. In the same timeframe as Fail2Ban has has reported vulnerabilities, OpenSSH has had at least 60: https://www.cvedetails.com/vulnerability-list/vendor_id-97/p...

"Worse than security theatre" is quite the statement given they reported and fixed those issues in timely fashions.

If you apply the principles of defense in depth, using the network layer to deny access to misbehaving remote hosts is an obvious win on a lot of fronts and hardly qualifies as security theatre anymore than using a network firewall is security theatre.

If we limit the use case to a single service, fail2ban is just a log cleaner. What it's detecting is merely the service that you're protecting doing its job properly. Now if you analyze the collected data and do something smarter with it, that's another story.

It's not 9 vs 60, it's 9 vs 0 if you don't use it, with no loss in functionality. And GPs point is that it's not defense in depth, the vulnerabilities in Fail2Ban can compromise the security of other layers.

I agree that almost all use cases of fail2ban are little more than feel-good exercises.

Failed login attempts (the noise) are not where bad things happen. What we should be concerned with is if the attempt succeeds but is not from a legitimate user. fail2ban is no help there.

Having said that it might be a decent way to collect IPs. At one point I was distributing the collected IPs from VMs and blocking them for the whole network. fail2ban does provide mechanisms to do this.

Absolutely agree. Fail2ban, sooner or later, bans you from your services becasue something in the configuration went wrong.

It does not protect against anything serious: you must have proper credentials/MFA or certificates and therefore bots can check as much as they want.

There is no protection against DoS either.

And I agree about moving the port - I only see a tiny activity in my logs coming from bots when my ssh port moved away. Obviously 443 is there to stay (this is a public service) so I will get whatever comes.

Related. Others?

Brute.Fail: Watch brute force attacks fail in real time - https://news.ycombinator.com/item?id=36169954 - June 2023 (259 comments)

Ask HN: How to protect against endless SSH login attempts to my server? - https://news.ycombinator.com/item?id=34077205 - Dec 2022 (27 comments)

Currently on the front page: https://news.ycombinator.com/item?id=37799438

At a former employer I hooked up a random Debian box to the corp network. Of course I ran fail2ban, in addition to a host of other lockdown measures. Without fail every morning at the same time some random corp scanner would start hammering on the ports, and fail2ban would kick in and block the scanner. After the default expiration time the rules blocking the scanner would expire, and it would happen again the next morning. This probably continued going on for several weeks or perhaps even months after I left the company.

You would get seriously yelled at by my team if you did this in my company.

The corporate scanner are not there for fun or because security loves to do scans. They are there exactly because they are random wannabe sysadmins who will plug in a box that ends up compromised.

A scanner may help to locate it and possibly warn about the issue. And only maybe.

Do not make life difficult for people who already have they hands full of constant vulnerabilities, fighting corporate crap and being the bad guys because they audit stuff.

We have a life too.

I was first exposed to fail2ban by some PBX server suite I was setting up. Totally cannot remember the name of it, but it checked to see if the fail2ban service was running. If it was not the PBX service would not start and throw a are you insane error message. Saved a lot of new kids like me at the time who were just fooling around I bet.

Same, VICIdial introduced me to it!

I really wanted fail2ban for my windows RDP. I'm using non-standard RDP port and still getting 10,000+ brute-forcing requests per day.

Found this alternative:

https://github.com/DigitalRuby/IPBan

What port do you use? Obviously you have to use one which is not common and not in the list of scanned ports of services like shodan or censys.

I have RDP on port 8443.

On another host I tried port 443, hoping to disguise it to appear like SSL. No any difference.

Also, there are services, which already publishing all my (and yours) open ports. Here is the report for my IP:

https://search.censys.io/hosts/104.63.172.143 (It's public anyway)

8443 is very common, id use something like 46434

This is ill-advised. Don’t try and hide it. Don’t publish it in the first place.

Both are useful:

https://github.com/EvanAnderson/ts_block

https://github.com/Zeziroth/Cancer2Ban

I made ts_block. It solved a particular problem I had at a particular time. Nobody should be doing RDP across the Internet w/o a VPN but, apparently, people still do it.

Shockingly it still works fine on Server 2022.

THANK YOU!

I have friends of mine that slowly migrated to ZeroTier for RDP.

In the world of amazing fast vpn solutions why is your RDP port open to the internet?

In security circles RDP is called “ransomware deployment protocol” - basically one should never expose it to the internet.

Make yourself a vpn box or some Linux with ssh and do port forwarding and allow RDP from that Linux host. OpenVpn or ssh are much better to be exposed to the internet.

I use wireguard for this stuff these days. Life is simpler now :) .

Be careful not to ban yourself :) had to use linode emergency shell once after misconfigured fail2ban blocked my ip

That's just a rite of passage we must all do.

The pro tip is to just not use it for SSH

To expand on this, don't use it because a good approach is to disable password authentication on SSH and use keys instead.

If for some reason you still needed password auth enabled I'd be inclined to still use f2b.

Every time I spin up a box by hand (which isn't often), I'll set up a sudo user with my SSH key, then drop out of the root shell and log back in as the new user. Only then will I disable root login and password auth over SSH and start setting up fail2ban and the like.

If I lose access by, say, switching to a new computer and losing my SSH key, then I'm more or less dead in the water. But it's a small price to pay and 1password supports SSH keys first-class now.

at that point, why not just do FIDO2 with a YubiKey or something? eg https://www.ajfriesen.com/yubikey-ssh-key/ as long as you don't lose the YubiKey or the backup YubiKeys you're good.

I have one but getting it to work over WSL2 is a challenge

I require both key and password to login, and have fail2ban rate limit password attempts.

This gives sufficient notice to fix things if a key were to become compromised.

I use it with ssh and password auth disabled, is there a reason not to? Might be overkill but the host is in my home so physical access if I ever get locked out is not an issue.

I think this is still reasonable, attackers may have a database of leaked keys (e.g. if you ever accidentally commited to GitHub, or ever ran a malicious script which uploaded it), which they then try on random servers.

Also be careful that you've used CSRF and other origin protections. Otherwise competition can DoS your legit users by rapidly cross posting via js from their own site. Not likely in enterprise world but could happen in less than ethical industries.

you can probably use a vpn to log back in

Works great, been using it for a long time to stop bots crawling sites. Also on hosts with open ports, like ssh that you cant whitelist.

Can this be a dynamic whitelist or is there an API to whitelist/de-whitelist specific IPs?

It's a dynamic blacklist. Any abusive IPs get banned for x minutes.

You can white list specific IPs though

> You can white list specific IPs though

This is what I mean, if this list can be dynamic or modified at runtime with an API.

The best friend of the small business LAMP specialized dev! Those mass scanners really love wordpress

There is a way to use fail2ban together with nginx's rate limiting to implement a temporary ban on addresses that are repeatedly abusive. As far as I know this is one of the least friction ways to achieve that. The limit_req directive works by dropping the responses that exceed the limit but does nothing more than that. Fail2ban can be scanning the nginx log files for instances of someone exceeding the limit, and add them to e.g. a Redis set. You can then use a Lua plugin to track incoming request addresses and check whether they belong to the "banned" set (then you can drop the connection fully).

Shoutout to my favorite docker image that leverages fail2ban https://github.com/linuxserver/docker-swag

For SSH I prefer blocklistd[0], which sadly is only available on FreeBSD and NetBSD AFAIK, it's a much simpler approach though requires a small patch to any daemons that want support.

[0] https://youtu.be/fuuf8G28mjs?si=UhuRVndacryMim_a

I came to mention this. I was impressed when it was announced and disheartened to see it ignored

Another couple fun things for SSH are iptables string module [1] and just adding a really long version addendum. [2]

Silly examples but the bots get confused. I admit these are not efficient rules. Try it out on that domain. Send some SSH bots my way, I honestly have no idea what it looks like on the bot-end-of-things. I would estimate about 95% of bots are libssh. About 4% are Go and the rest are an odd mix. That string can be spoofed but I think most people are too lazy.

[1] - https://bender.ohblog.net/temp/ipt.txt

[2] - https://bender.ohblog.net/temp/vers.txt

I forgot to mention, if the IPtables examples are useful they can make things ever more quiet if instead of "-j DROP" one uses "-j SET options" to add to an ipset, then use that same ipset in an outbound rule to tcp-reset the openssh socket to free up memory faster. Not really a big deal unless one gets flooded but probably useful to someone some day. Maybe I will do a little write-up on this.

You can achieve something similar with just iptables / nftables that doesn't involve parsing logs.

https://blog.bigdinosaur.org/securing-ssh-with-iptables/

The first thing you need to do on any new Linux server installation:

sudo apt install fail2ban # Choose your flavour

And you get rid of most pesky SSH knocking traffic and credential stuffing attacks.

Changing the port to anything other than 22 does this as well.

Changing the port reduces the traffic, yep, but definitely doesn't get rid of it - all my servers use non-standard ports for ssh and there's still ~10 login attempts a minute.

2244 ? I used to get almost no tries on 2244 which is kind of an obvious choice. Nowadays I have wireguard and no ssh ports exposed as they run on private IPs.

Don’t ban hosts or any service account automatically. Use conditional accesss, mga for hosts with credentials + IP or client cert. Auto raise a snow ticket for the support team and an alert in your SIEM for your Soc. If you want more detail I’ll put my blog in my profile.

I personally would rather recommend a whitelisting functionality, using a trusted third party.

I wonder if SSO services can provide IPs of successfully authenticated against their services users.

Jump hosts are a well-tested and widely-deployed pattern already that basically do the same thing without the downside of lockout problems from connecting from somewhere new.

I was recently thinking on a way to open up my Jellyfin Media Server, so I can stream music from my personal collection. Jumpbox would easily incur a lot of traffic costs... due to all of the media being very large(many users of jellyfin suggest that). I also would like to allow friends access to my library, without them having to connect to my VPN.

Having had a very bad experience with getting my home network once DDoSed and once hacked - I am at a stage where a reliable third party source, that can tell me what IP to whitelist, is the only reasonable solution I can come up with. (I know it's not a common use case, but it is my use case)

Probably needs a date, since this tool is extremely old?

ls and cat and ssh are also old.

Why was this posted? It's a great project that's widely known and pretty much ancient in terms of OSS projects.

A lot of people might not know about it yet, and genuinely find it useful

https://xkcd.com/1053/

I would guess the subtext here is related to the 23andme breach.

The problem with fail2ban is that an attacker who has a botnet of significant size at their disposal won't even be slowed down. Not saying it's worthless, but it's not a silver bullet.

Not saying fail2ban is in any way state-of-the-art security or even best-practice at this point, but if they're burning an IP after a few bad attempts, that's going to be more effective than letting them try tens of thousands per IP, no? Security in layers of course, so even a few [dozen, hundred] failed SSH attempts should probably raise an alarm or put things into an elevated security mode automatically...but also don't leave SSH open to the public, especially not on :22.

> but also don't leave SSH open to the public, especially not on :22

100%. These days, at least if you're working out of a cloud provider, there's no excuse for exposing SSH to the world on any port. AWS/GCP/Azure all have different tools to allow you to run bastion-type services without internet-facing SSH.

I like occasional discussions on these things. People sharing tips, experiences, pitfalls, etc. It gets everyone in one place at around the same time to do it, like a small digital convention.

Since IP addresses are PII, how can I use this and still respect GDPR? It's not like I can use consent for processing.

Since you're also obligated to secure said data storing the IPs from brute force attempts is not only fine but almost a requirement. You're not making the connection to any individual and the use-case is legitimate.

You don't need consent because this information is strictly required for a legitimate business purpose (ensuring machine security)

This is explicitly made an example here: https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-re...

Firstly, the GDPR doesn't care about PII - that's an American legal term and the GDPR is an EU law. It deals with processing personal data, and the two are not the same (although there is overlap, with PD having a more expansive definition).

Secondly, consent is only one of six legal bases for processing personal data. The other five are that:

- processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract;

- processing is necessary for compliance with a legal obligation to which the controller is subject;

- processing is necessary in order to protect the vital interests of the data subject or of another natural person;

- processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller; and

- processing is necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data.

The last of those - legitimate interests - is clearly appropriate, the rights of the data subjects (i.e. the IP addresses' users) don't override the legitimate interests of securing one's machines, especially given the limited processing performed by tools like Fail2Ban.

Actually, GDPR doesn't forbid using PII for legitimate interest like fraud detection

Reply on news.ycombinator.com