Hacker News Reader

Comment by tptacek | Hacker News Reader

In a sane configuration, the only thing a tool like this should be doing for you is keeping your logs clear. If you're deploying something to mitigate the risk of brute-force attacks on your server, the problem is that it's possible to brute-force your server, not that you don't have an autoblocking system enabled.

These tools are popular, but I think they're kind of silly.

Later

I read some comments below, which compared this to Fail2ban (I had written this tool off as yet another Fail2ban-style script). But it's even scarier: it's written in C. I guess this is from like 2010? Don't run stuff like this.

View on HN

I agree. I never understood why people use all this stuff. If you use strong enough passwords, it's impossible to be brute-forced, simple as that. If you can't control people on your servers and afraid of weak passwords, just disable password authentication and that's about it.

Now the issue of logs is real. I tried to configure debian to remove noise from the logs and I couldn't. May be someone with more knowledge would suggest how to do so. Because I'd like to see successful authentications in the logs and only those. So that could be a valid reason to change port, etc. But that's only because logs are lacking proper configurability.

> I never understood why people use all this stuff ... Now the issue of logs is real. I tried to configure debian to remove noise from the logs and I couldn't.

Sounds like you understand at least one reason people use stuff like this (including, most prominently, fail2ban). Rate limiting and banning IPs trying to brute force and/or probe open ports (not just SSH) goes a long way to cleaning up your logs, your real time state, as well as other things, like frustrating the attacker, not wasting connections on them.

Having said that, this particular tool looks a bit shit ...

> Rate limiting

You don't need fail2ban to rate-limit SSH.

RFTM ... MaxStartups, PerSourceMaxStartups, PerSourceNetBlockSize.

> trying to brute force

     AuthenticationMethods publickey
     PasswordAuthentication no
     ChallengeResponseAuthentication no
     PermitRootLogin no

Not disagreeing in general and disallowing password login for root is IMO really a no-brainer.

But, it's nonetheless a fact that a (temporary) nft drop rule in the kernel is much more effective than traffic hitting a user space daemon like SSH.

Some NICs even allow offloading that (e.g., in form of a bpf program) to the card itself, so that such traffic won't ever hit the system (CPU) at all.

So yes, SSH can be made secure and rate-limited without anything else, but that won't be the most efficient solution; now fail2ban/sshguard/... ain't no free lunch either, so one needs to weight the added (IMO small, but still) complexity against benefits in a specific use case.

> (not just SSH) ...

We know about the SSH daemon config. You still get all sorts of probing that isn't a brute force on SSH.

Exactly, fail2ban/blacklistd is for software who lacks certain function ssh has for example sendmail, asterisk etc:

https://github.com/fail2ban/fail2ban/tree/master/config/filt...

Just to double check my sshd settings I logged in to a server I have, and while it had "PasswordAuthentication no" set in the normal place in the config file, there was a line added to the very end of it "PasswordAuthentication yes", which I guess overrides the previous one.

The server is reinstalling the OS right now...

FWIW, your guess is incorrect. From sshd_config(5): "For each keyword, the first obtained value will be used."

Still, if you cannot explain how that second line got in there, it is probably prudent to reinstall (perhaps even reflash BIOS).

I primarily use it for logs.

However, I've worked at a lot of startups, and no matter how much you try to discipline people, someone thinks their circumstance is special (and it somehow requires a password-based login), or someone is just lazy and sets a password for unfathomable reasons. So, for those people, I run fail2ban to allow it to act as a rate-limiter. But tptacek is right, in any sane situation, you don't really need it. But sometimes the situation isn't sane.

> just disable password authentication and that's about it.

a.) for what ever reason, I've had third-parties insist on passwords. (In healthcare¹, too. Ugh.) b.) For quite a while, Azure's "AADLogin" used keyboard-interactive, which meant that you couldn't disable it. Thankfully, that got fixed in a recent version.

¹In addition to fail2ban, we also did firewalls at the cloud level. The password was a good (secure) one, but I'd sleep easier with a key-based login.

Big issue in healthcare is a lot of legacy integration brokers that had SFTP bolted on post-release using the existing UX and config framework for FTP, which had historically been the dominant protocol in healthcare. Depending on the broker sometimes key auth isn't supported at all, it is supported but has to be configured in a strange way that's different from the normal SFTP/FTP config, or it's supported just fine but the company has business processes like paperwork for their integration subcontractor that still assume FTP is used (this latter one does unfortunately come up...).

You'll see the same thing in financial software for basically the same reasons.

> I tried to configure debian to remove noise from the logs and I couldn't.

Not very elegant (as in, not with built-in tools), but using Logstash would be one way of achieving that. There probably is a better (simpler) way, though.

https://www.elastic.co/guide/en/logstash/current/pipeline.ht...

Edit: After some research, rsyslog seems like a more appropriate candidate. But the docs don't seem too novice-friendly and the configuration is quite obscure. Tradeoffs.

https://en.m.wikipedia.org/wiki/Rsyslog

I don't think anybody with a sane config need fail2ban or similar. But as a sysadmin who works for clients who don't understand such concepts, sometimes it's easier going "okay, we will install it on your systems for $x" than explaining why it's useless. It's a business politics tool.

apt-get install logcheck

Have you looked into logrotate?

> If you're deploying something to mitigate the risk of brute-force attacks on your server, the problem is that it's possible to brute-force your server, not that you don't have an autoblocking system enabled.

Well, even when accepting only certificates some brute force bots are dumb enough to keep trying to connect to try passwords, and this can give a lot of unnecessary CPU load due to all the initial connection crypto handshakes.

Though I have to say most of the bots these days are smart enough to quit as soon as password auth is turned off. But 10 years ago most were really dumb. Using fail2ban was more of a DDOS prevention than a security thing.

It seems improbable that the baseline load of a couple concurrent SSH attempts is higher than the load of running an extra Python daemon that's constantly checking your log files.

When I still ran my own server it was a pretty heavy load, like 30% or so. Much more than a minor nuisance.

But it was a SPARC server with 1 core at a few hundred MHz and no crypto acceleration. As you can tell this is a while ago :) So yeah I guess these days it's much less of an issue. But the number of possible attempts will have increased too of course.

I figured I might as well go pull up some numbers from my personal systems. I've got 3 internet-facing VMs that expose SSH to the world.

For the past 30 days, I've seen a combined total of 86k SSH attempts. I've not bothered to filter out my own legitimate sessions, given that they're in the ballpark of ~3 connections a day tops.

Total load on each of these systems has stayed below 0.5, and that's including the other actual work the servers do. I don't have a great way to pull out sshd specifically, but it's safe to say that these servers haven't had their operations affected at all by the nefarious attempts.

For what it's worth, the one thing I'm doing that could potentially impact the resource consumption is hardcoding cipher/keyexchange/MAC selections. Given that I control 100% of the legitimate clients for these SSH daemons, I've hardcoded the settings as follows:

Ciphers chacha20-poly1305@openssh.com

KexAlgorithms curve25519-sha256@libssh.org

MACs hmac-sha2-512-etm@openssh.com

HostKey /etc/ssh/ssh_host_ed25519_key

Over 95% of the spam connection attempts get dropped before my server ever does any crypto, because the connecting system doesn't support that KeyExchange choice.

> I don't have a great way to pull out sshd specifically

If it's using systemd, try

    systemctl status sshd

If CPU accounting is turned on, you'll see how much CPU time was used by sshd, plus when it was last restarted, which at least gives you something to work with.

If not, you can enable it with:

    systemctl edit sshd

then adding:

    [Service]
    CPUAccounting=yes

and restarting the daemon.

TIL about systemctl edit, thanks. Note to others: if you want to override an existing setting, rather than just add new settings, you'll need to use 'systemctl edit --full'.

This is really useful, thanks!

TIL! Thanks for sharing

sshguard is C, and the log file for sshd won't have very much output if there are only a couple concurrent SSH attempts. Don't use this, but not because of concerns about performance overhead.

From the comment I replied to:

> Using fail2ban was more of a DDOS prevention than a security thing.

On the other hand, shouldn't we be stealing resources from bots, even if it costs us resources too?

No. Let them bother someone else :P

I _WANT_ to harvest IP's of PC's trying to SSH into any of my servers. I want them because then I can block ALL traffic coming from their IP's.

At my last gig, I had thousands of domains and hundreds of servers running on every major (and a shitload of minor) service provider. I custom scripted all these servers to compile all the IP's that attempted a connection, send those lists back to HQ, and then distribute back to each node a block-list table that I could dynamically apply to each edge device. One compromised computer even trying to login was completely blocked from 5000+ domains within 15 minutes.

I'm not worried about my ssh login being brute-forced, but if your computer is compromised I want nothing to do with you.

You might be interested in a side project I've been working on...

While its similar to sshguard or blocklistd, I didn't know they existed when badips died a couple years back. https://www.nubi-network.com/faq.php

The list file should work for either ingress or egress (or both).

I wanted to post about it on here some time ago, but I'm kindof bad at advertising.

There is a whole Internet worth of random IPv4 addresses being variously used to variously attack various computers. Your blocklist doesn't even approximate it. What's the point of trying to track and block 0.000000001% of the machines that might be used to attack you?

What’s worse, those IPs are probably (at least in part) dynamic, from VPS servers in data centres, or using residential Proxies or compromised machines behind a NAT. A soon as those addresses change or get assigned another customer, you’re blocking potentially legitimate traffic…

I dont see anything wrong in collecting and blocking them for some time, not infinitely of course. Blocking at the firewall effectively means blocking further communication and requests that cost you bandwidth and computing time. Why not save that for the legitimate visitors?

Because you block legitimate visitors. After we ran out of IPv4 space and started to use CGNAT instead of IPv6, the internet has become a bastardized version of itself.

If carriers/ISPs monitored their networks better (or acted on abuse@ emails), their IPs wouldn't wind up on blocklists (either temporarily or permanently).

edit: That isn't to say they should be logging every packet... But if I work for XYZ Hosting Company and spin up a new VM, hand you the IP, and you turn the IP into a mini-shodan scanner... Is the hosting company at least a little responsible for what happens next?

I want that for servers I manage.

I’d say that fail2ban is a Band-Aid too, if brute forcing passwords is an issue your problem is that you’re not using key based auth.

If someone wants to go to the trouble to break crypto, it would be cheaper to infiltrate the ssh project and implement back doors after a few years of building trust. If that’s your threat model, double wrap your protocols (WireGuard jumphost > ssh type stuff)

As you correctly point out, password authentication should not be used for SSH to begin with, but more importantly fail2ban is not a piece of software written to stop brute force password attacks, it’s a very general tool for triggering network blocking actions based on specific patterns found in logs.

My only point being that there are some great use cases for fail2ban, and while this isn’t one of them it’s important to remember that this does not make fail2ban a misdirected or useless security initiative by default, in case anyone gets that impression from reading these comments.

It doesnt stop determined attackers who have access to a botnet, but it does stop low sophistication attacks.

If brute forcing anything is a concern then addressing that concern should come first, focusing effort on fail2ban is a cost that is probably better spent elsewhere.

Doesn't it make sense to do both given that the cost of additional mitigation is trivial?

No, and to elaborate, fail2ban has had an RCE vuln in recent memory. These things add attack surface, which is the opposite of what you want.

No.

Remember when fail2ban had an RCE to execute things on a server via root user and it was as simple as parsing an email string because it was executing the mail binary on the other end? [1]

That's pretty much the reason why I started to write my own guard tools.

I don't need people from China or Russia logging in to my server. I don't understand why tools like this are never built in an ASN aware manner, blocking mechanisms would be so much easier.

[1] https://research.securitum.com/fail2ban-remote-code-executio...

> write my own guard tools.

if (notMe)

    panic();

Though I agree, security is a mindset not a tool you can simply download or run.

  But it's even scarier: it's written in C.

I don't want to defend this tool, but OpenSSH is written in C as well :).

OpenSSH doesn't try to parse untrusted text input that can be fairly easily manipulated by an attacker (i.e. in this case the logs). That's something that should be scary in any language, but C is definitely one of the worst choices possible for this particular task.

Two Cs don't make it Rust

Possibly the best tool in this space is blacklistd / blocklistd, which just exposes some reporting APIs and hooks them directly in sshd, rather than trying to parse failed logins out of logs. That said, meh.

https://github.com/zoulasc/blocklist

Edit: Yeah, there's no race condition; the client sends the offending socket to blocklistd over the blocklistd socket.

(Originally…)

I'm a bit curious; it also seems to imply that it logs the fd number, and then blocklistd queries the kernel to obtain the remote's address. That seems like a race condition:

  1. blocklist call is made
  2. daemon closes the socket (connection has failed, after all)
  3. next client connects
  4. lookup is made by blocklistd, and the wrong peer is potentially incriminated

(Actually, perhaps it's sending the fd over the socket. I forget you can do that, and the README is hard for me to follow. The "like syslog" also implied that it's logging over syslog to me, but that's not what the code does.)

>These tools are popular, but I think they're kind of silly.

Really? And how do you block someone who slowly try's (every 2-4 minutes) to bruteforce your smtp? Yes your openSSH has some slowdown mechanisms builtin, but your Imap-Server? Smtp-Server? Your openTTD-Server? However i think using the logs is the wrong approach hence my use of blacklistd* whenever possible.

* https://www.freebsd.org/cgi/man.cgi?query=blacklistd&sektion...

* https://man.netbsd.org/NetBSD-7.0/blacklistd.8

Hi Tom - I don't know if you've already written about this elsewhere, but what would be some sane defaults for SSH?

Do you think using key-based auth and disabling password authentication is enough? or are there other things you would recommend?

OT - nice Fibonacci in your profile :D

I’ve always used this: https://www.sshaudit.com/hardening_guides.html

If there’s something better please correct me!

I wasn't familiar with this.

I find it interesting that one of the suggestions is to disable the ECDSA host key. I thought ECDSA was one of the "good" algorithms?

Its suspected from the snowden docs that this is one of the ones intentionally weakened by NIST.

Here's the long answer: https://blog.cr.yp.to/20140323-ecdsa.html

Nobody serious thinks the P-curves, which is what we're talking about, were weakened by NIST. I don't think anybody at all thinks the DSA algorithm was "intentionally weakened", but lots of people, reasonably, believe it was incompetently designed.

Ed2519 has the virtue of both a better curve and a better signing algorithm, which is why it's used in preference to ECDSA. If you're using 25519, don't bother keeping the rest of the algorithms enabled.

I think these hardening guides are mostly pretty silly too, for whatever that's worth. The important thing is making sure nobody can log into anything with a password. The rest is just liturgical.

I think it is the issue where ecdsa is a NIST standard that was based on an elliptic curve provided by the NSA. As the NSA has been known to suggest poor cryptography primitives in the past, many groups recommend ed25519 (same process, different and more trusted curve generation process)

Take a look at the "concerns" section of the wiki page. https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signatu...

Wow, that's terrible. Imperative instead of declarative, and does not explain the reasons. Perfect for cargo-culting your way to disaster.

Also, why isn't hardened ssh the default? Is this breaking too many things for the user?

Compatibility basically.

Would you recommend a better guide? Is the config suggested by it bad? Not everyone has the skill/time to deeply understand everything immediately, and having a saner starting point is a good thing.

while it is not the best, I do regularly code audit OpenSSH.

For sane default, you can generate your working but sane ssh_config and sshd_config based on my inner code knowledge and network security and as an IDS/IPS architect.

Each and every known OpenSSL (v8.4; v9.0 is underway) settings in the config files have annotations and details and many have additional links detailing why. I also note where each and every config settings are found in the source code by nesting, what protocol state, control state, authentication stage, and lockings (makes code review so much easier for me) also in its comment section of each config setting.

Even has a bash script to let you create these config files (defaults to your subdirectory for pretesting, but can also as an option update /etc/ssh, which I confidently do) into something that would pass an SSH audit for ssh-audit (I’m a contributor), CISecurity and often better.

Also these scripts generates a script containing proper file permission settings based on top generic Linux File System variants (basic variants like APT, pacman, Portage, DNF/RPM) ).

I also offer SSH bastion setup (adjacent to my URL given below) as well.

I also enjoy using certificate-by-user as an authentication mechanism for maximum ease of sysadmin use when it comes to emergency mass-blocking or occasional employee departures.

https://github.com/egberts/easy-admin/tree/b74765baa450593be...

I have it installed, but I'm not sure I understand why its as you say.

Why are you scared of C? Also, should I remove sshguard? I dont want my server going down.

Btw, it has been updated after 2010. Are you talking about 2010 release ?

I mean, yeah, these things are kind of silly. So is, "But it's even scarier: it's written in C. I guess this is from like 2010? Don't run stuff like this."

Fail2ban is written in Python and managed to have a vulnerability.

Since when being written in Python automatically makes a program less vulnerability prone than usual?

It depends I guess on where you start the count; some people would say Elias' "Smashing The Stack", I'd say Lopatic's HP-UX exploit, some people would go all the way back to RTM's worm.

Well it does make the developer try harder to have a vulnerability. You can't just forget to bound check an array indexing, you need to actually take instructions from the Internet (e.g. eval, import, or exec in case of fail2ban), but of course the way this happens is usually very indirect (e.g. race condition).

No programming language can stop a determined developer from creating a vulnerability, but some languages make the task easier than others.

It's mostly shell scripts, with a few C handlers. Very minimal.

Defense in depth, cost little to add one more layer.

Security always works in layers, it should never be just one line of defense ...

Even obscurity has its place (e.g. using a different SSH port)

> But it's even scarier: it's written in C.

Why would being written in C make it scarier?

Two terms: undefined behavior and memory safety. Try searching the web for those terms.

The question wasn't about common mistakes C programmers make

Reply on news.ycombinator.com