Hacker News Reader

Comment by throwaway613834 | Hacker News Reader

Question: I've had an SSH server open to the world for months on a nonstandard port. I've never gotten any attacks logged on fail2ban, to the point where I suspect fail2ban is either not logging the attacks or not operating correctly... even though it logs every login I'm aware of just fine, so it really seems like there really haven't been any attacks. I find this so unlikely. Is this normal? Does no one scan high ports ever? Are there any more plausible explanations?

View on HN

Is this server running a recent version of Ubuntu and using systemd? Also, have you verified that fail2ban actually blocks you by intentionally making bad login attempts to trigger a ban? I do this with every new machine and never assume it works until I can trigger it on demand. (test using a VPN or set a very short bantime so you don't lock yourself out)

I ask because systemd may not be writing the failed login attempts to the log file fail2ban monitors, which means fail2ban may not be responding to attacks. I don't recall exactly how I resolved this, but I think I had to set "backend = systemd" in my jail.local under the [DEFAULT] section header.

Oh wow, thanks a ton for mentioning this. It's Ubuntu indeed but I seem to have sysvinit rather than systemd from what I can tell. I have no idea how to actually test it in a way that ensures I will trigger it though (since I don't really have other setups that I can test the same method on and ensure I'm doing it correctly). What do I have to do to trigger it? Just fail a bunch of SSH logins in a row? (I have ssh, dropbear, pam-generic, and ssh-ddos enabled.)

> What do I have to do to trigger it? Just fail a bunch of SSH logins in a row?

Yes, that's what I do. If you have SSH login via password disabled (you should) then just temporarily move your private key(s) out of your ~/.ssh directory and try logging in a few times. "ssh user@machine && ssh user@machine && ssh user@machine" will do it if you have a ban set for three failed attempts. If it's working, your target machine should no longer respond at all, and connections will time out. If it's a web server the site should be unreachable with your web browser until after the bantime passes.

Thanks, I'll give it a try :) I've gotten myself locked out before like that, but ironically from a different machine...

Ack, I figured it out. The ban does get triggered, but it doesn't succeed because iptables doesn't work on my system -- some module isn't properly compiled for the kernel. Thanks for helping me figure this out!

No problem, and this is a good reminder to test critical systems periodically. Same goes for backups! Good luck.

No, this is pretty expected. All the script kiddies are scanning port 22 and nothing else. Recently I made the mistake of changing a root password (I know, SSH root acc ss, bad idea) to a dictionary word. In 1 minute the system was penetrated.

With a nonstandard port, bots mostly don't even bother to scan you.

Interesting, good to know, thanks for confirming. I guess security by obscurity is quite effective!

Technically, for an uninformed attacker, this is just one more level of entropy added to your password (multiply the combinations by 65635 - and then more in practice, as you have to account for the fact that this has a low probability of occurring from the viewpoint of the attacker).

It just really is a part of your password (albeit one that can be guessed easily by a more dedicated attacker, just like the username).

Right :)

It is not plain added entropy. You can't multiply the complexity (or add the entropy) because there is a signal when the attack makes a partial guess (he gets the openssh prompt).

The added entropy is a bit harder to calculate, but unless your password has 16 bits of entropy¹, it will round pretty well to zero.

1 - Why TF would you run a ssh server that accepts passwords anyway? But nearly everybody I have seen that thinks changing the port is a good form of protection also sees no problem on accepting them.

That sounds like a much better solution than I've used in the past. By simply restricting total # of connections / IP to one my attacks dropped dramatically, but not gone completely.

I think it's just a matter of numbers. The IP search space is already pretty large with IPv4 on just a single port. With IPv6 it'll be more or less impossible for attackers to make any substantial progress by just searching for IP addresses to attack, let alone an IPv6 address on a non-standard port.

After reading this I'm likely going to be upgrading my technique. SSH on IPv6 only (eventually shutting down IPv4 altogether, can't wait), listening on a non-standard port, limit 1 connection per IP.

Yeah, definitely listen on a nonstandard port. I'm not sure how to limit to 1 connection per IP but it sounds rather limiting, so I wouldn't do it... I often have multiple SSH sessions from my own IP.

I just run a 'screen' session to open 10 bash sessions simultaneously when I want to do anything substantial. That has never felt restrictive to me.

Can you see two screens simultaneously with screen though?

I'm not sure. The need has never come up for me though.

If you want this, I would suggest tmux. Tmux by default comes with tabs (multiple terminals) and panes (multiple terminals visible at once within a tab) where you can split your screen vertically and horizontally however you like.

Usually, you want one full-height terminal and two about half the height of your screen.

http://www.kaltpost.de/wp-content/uploads/2013/02/tmux.png

But these are so much more restrictive. Right now I just open two terminal windows -- they can overlap each other, I can make it so one covers every part of the other except a tiny part whose update I want to see, and make changes in these first one. I can kill programs in one without being worried about accidentally killing them in the other. I just don't see the appeal of forcing everything into a single terminal screen honestly. What's the benefit?

If your network connection drops long running jobs keep running, no setup.

I do most of my work on our university cluster over SSH. I can close/shutdown my laptop at any time (or have my connection drop) and as soon as I SSH back into the server all my editors, shells, running jobs, etc. are exactly how I left them.

Oh, that's not quite what I meant. The question wasn't about whether I should use {screen,tmux} to open a shell; the question was whether I should use {tmux,etc.} to manage multiple shells in the same terminal window. As opposed to having multiple instances of them open in different windows.

Mosh is also worth a look.

You want ControlMaster. man ssh_config.

You can also share an ssh connection between sessions.

Give yourself some breathing room. I limit connections to four in a three minute window.

    iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 4 --rttl --name SSH --mask 255.255.255.255 --rsource -j DROP

Don't know if it is a too obvious answer, but have a look at the output of 'sudo lastb', it will show failed attempts quickly to verify if it is plausible and fail2ban properly setup.

I've never heard of the command, so it's not obvious, thanks. However, I'm not sure how to interpret this. All I see is a single line:

    btmp begins Fri Sep  1 06:25:05 2017

Does this just mean nothing has happened? I'm not sure what's special about that date; the setup has certainly been running for much longer than that...

That's not a bad sign then. Mine shows lots of attempts. lastb shows the last unsuccessful (bad) login attempts. it uses the file /var/log/btmp as input. the time stamp of "btmp begins .." either is the same as the creation date of the btmp file or if the btmp file was rotate of an older version i.e. /var/log/btmp.0

I've been running non-standard ports for a few years and while it certainly cuts down on the non-stop brutes on 22, my servers still get smacked pretty often in 400-500 attempt blocks.

Oh, interesting. Are your servers targets of interest somehow? Or are your ports possibly still low numbers?

definitely not interesting, but they're on port 8022 so maybe that's a common one for attack scanners. i mean with masscan its so fast and easy to find ssh ports, i cant imagine any non-standard port is more obscure than the next. they are hosted on a low cost vps provider so maybe their ranges are common low hanging fruit.

I wouldn't have revealed the exact port number here, but thanks :) yeah, I think your explanation sounds plausible.

A port number isn't a real secret, it just screens out the bulk of mass scans. There's no point trying to hide it from targeted attackers.

The hosting providers have known IP ranges and they are scanned permanently. I wouldn't be surprised if some guys had a rule to try 8022 and the most common non standard ports.

Reply on news.ycombinator.com