What SSH Hacking Attempts Look Like
medium.com
medium.com
It's basically a clone of Mozilla's, except enforcing only the very strictest encryption. You can just lift the first 16 lines out of that and merge them into your sshd_config if you don't want to change your existing sshd_config too much.
I can confirm from the same experience, however, that it prevents most hacking attempts from ever going to the authentication stage.
- Have a stand alone SSH server that is something like a R-Pi[type B] running an OS that gets patches regularly via unattended-upgrades and reboots itself at least once a week.[could also be a minimal VM like AlpineOS if you need Gbps+ line speed]
- Have this R-Pi and your network gear plugged into a UPS that can withstand at least a couple of hours of power outage.
- Use non-standard ports on your perimeter FW for forwarding to the R-Pi.
- Run fail2ban or something similar. Set a bantime of at least an hour[3600 sec] after no more than 5 attempts in 600 sec. This is mostly to discourage anyone who stumbles across your perimeter listening port.[which enough will] Password login will be disabled anyway, but a bot might not check for allowed auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>]
- No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no]. Use public key based authentication.[PubkeyAuthentication yes] Put the public SSH key[id_rsa.pub] of any machine that you wish to authorize into ~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-authorize remote machines[with particular users] at will.
- add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config
- No root logins allowed![PermitRootLogin no]
- Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed to access to SSH.[whitelisting]
- Periodically review your logs of perimeter FW, R-Pi FW and /var/log/auth.log to see what's going on. It should be pretty quiet, but look at them anyway once a week.
- Sleep well knowing that you have a simple, layered and robust defense strategy that also has power-event survivability.
Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds.
Don't use your perimeter device to host SSH services. It will not get patches fast enough when vulns come up.
I don't know how much it adds, but it is non-zero.
The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero (other than your own use).
Port knocking has no place in security by itself but I think it's a wonderful addition to a layered defense - my favorite one, in fact.
While I completely agree and encourage people to use all of your suggestions, this single step - not being on port 22 - has easily provided the most benefit. When sshd listened on port 22, I regularly saw attacks that would run through thousands of common passwords at high speed[1]. After moving to a random port, it was over a year before I saw another hostile login attempt.
It may be easy to discover the server with a simple port scan, but in practice the attacks seem to focus on the easy targets.
> fail2ban
Fail2ban is great and highly configurable. I could accept having to manually reset occasional false positives, so my fail2ban will immediately (no retries) ban on the first use of anything that is known to be malicious such as any attempt to login as root with a password, asking to login with an account name like [admin, mysql, squid], or any apache access log matching /\/phpMyAdmin/, etc.
[1] Several logins per second over ADSL could use a large fraction of the upload bandwidth.
PasswordAuthentication no
^ it should return that. The end.
Imagine if you look for something else, you have to point the cursor in the middle, right after grep and before the file. With cat |grep you press Arrow-up and Alt+Backspace and you are right where you want to be, ready to search for something else.
In scripts you can save the extra process though.
I would find it okay without cat to start the next line with "grep newPattern " and then fetching the previous filename with alt-. (edit: or !$ of course, but alt-. is more practical) -- not saying it's necessarily always better, but knowing your shell better (and/or using a better one) is often a viable alternative to using another construct.
Also, than* (sorry)
I don't see how the rest is faster than using Arrow-Up and deleting the last word. It seems more complicated. The most important string is still in the middle.
The shells were developed on keyboards that didn't have arrow keys and have affordances for them.
I doubt the later one.
The teletype was a printing terminal and the ADM3A was a “glass teletype” — essentially a printing terminal too.
It’s possinle to write a lot of good code this way
FWIW anyway Brian wouldn't have thought about those keys as the intended use of Readline would have been emacs keystrokes (emacs, or EMACS as it was in those days, had recently been ported to Unix by RMS).
Interestingly our 36-bit machines (PDP-10s and lispms) had arrow glyphs on their custom non-ASCII keyboards (KTV and space cadet) but they were for mathematical notation, not cursor movement!
Thank you.
It reminds me of the idiom of putting "# -- coding: utf-8 --" at the top of Python 2.x files. Which is incredibly useful, but nobody remembers, and googles everytime.
This comes from the fact the first famous tutorials on Python where written by emacs users, and this editor recognizes this idiom.
But fun fact, the regex matching this line is "^[ \t\v]#.?coding[:=][ \t]*([-_.a-zA-Z0-9]+)" (see https://www.python.org/dev/peps/pep-0263/) and accepts many variations (including a vim format).
The simplest variation is the dumb "# coding: utf8", which is not only straightforward and easy to remember, but way less magical for anybody reading the code.
Bottom line, if you are stuck in 2.x, use "# coding: utf8".
Of course once you use the 'cat' trick you then have to remember which comments require '-', '-f -' or something to that effect because whoever programmed them didn't think the primary use case was to work as a filter.
Pipes are one of Unix's most useful features and cat somefile | asequenceofusefulcommands is a shorthand that anybody should be able to use.
ChallengeResponseAuthentication noI'd recommend SSHGuard over fail2ban though, I seem to remember the version of fail2ban in the Debian repos completely choking on IPv6 and failing open which is obviously undesirable.
Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server. If it indeed always fails open then there's no reason for me not to use it.
Though, I don't know how it could be guaranteed to always fail open.
But it fails open whilst you think there's a layer of protection there.
> Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server
Most of these tools will let you whitelist an address - and presumably there's usually a way back in via a serial console/IPMI-KVM interface?
That's pretty easy to test for.
>will let you whitelist an address
I don't have a static IP address.
>usually a way back in via a serial console/IPMI-KVM interface?
I haven't been able to do this at my provider when password login is disabled. Maybe I just don't know the way.
Is using non standard ports a reasonable approach? On one hand, it's kind of a security-by-obscurity measure, and it's also a (very minor) inconvenience to real users. However if it's not being used in place of other reasonable security measures, I'm not really sure what's bad about it, but it does feel a bit janky to me.
Same goes for other non-public services such as VPN.
What ports should be used? I think clearly you don't want to use other well known services (eg, running ssh on port 25 is just going to fill your logs with failed connection errors). Should you use ports >1024? >10000?
I see it as a way to reduce the number of attacks, which is helpful, and may end up being more secure, but I wouldn't count on that alone. Against botnet attacks it will likely help, but against someone specifically targeting you, it likely will only slow them down.
Basically it boils down to 'why not?' for me.
It's a game of percentages.
Again, percentages. With enough attempts, some will succeed.
I can only speak for myself, if other people are ignoring warnings, it's their problem.
I believe that's been adequately addressed.
Restricting the remote IP range is another option.
This comes up as a point of confusion often, so allow me to clarify:
A non-standard port as the only security mechanism (e.g., an unauthenticated HTTP admin-interface) would indeed be 'security through obscurity', but a non-standard port as an extra mitigation layer on top of existing security mechanisms (e.g., SSH authentication) is 'defense in depth'.
A user can keep running a program to check if your non privileged port can be bound and if it could, it can start any behaving daemon that could be pretending to be a valid SSH daemon but could have anything like key loggers. Even a single restart of sshd could make it happen.
You say it yourself:
> also a (very minor) inconvenience to real users
I would disagree with the “very minor” part. To paraphrase myself (https://news.ycombinator.com/item?id=6617312):
As I understand the argument, it’s “Changing port number add security, therefore it’s a good idea.” I think nobody argues that it adds security. The problem is that:
1. It adds very little security: 16 bits is not much, and the result is not 256 bits (say) of SSH key plus 16 bits equals 272 bits, but instead effectively still 256 bits, or 256+8×10⁻⁷³ bits.
2. The security it adds is itself bad (sent in cleartext, easily brute-forced)
3. These problems stand against the many drawbacks of this previously discussed (complexity, confusion, etc.).
And the final argument: If increased security is what you want, simply increase your key lengths and/or password lengths, and you will get much more than 8×10⁻⁷³ bits of security, without any of the above problems.
Technical protection is likely limited, but then, admin attention is also a limited resource.
An entirely different effect has been brought up by others already; a non-standard port is enough to evade ~99 % of all automated attacks, therefore you can run at higher log levels and might notice actual attackers earlier.
Like I said, nobody argues that it adds zero security, but that, for all the hassle it introduces, it adds way too little security to be worth it. 16 bits is nothing, and can be brute forced quickly.
Regarding port knocking: I’m assuming that most people don’t write their own port knocking software, but use something standard. The configuration of any port knocking scheme is equivalent to an additional separate key of length log2(number_of_possible_port_knocking_configurations). What is the effective “key length” of port knocking schemes?
Also remember that the “key” of port knocking, just like a non-standard port number, is transmitted in the clear, so anybody listening to the traffic can see it. (And if you assume that the attacker can’t listen to the traffic, why are you even using SSH instead of something simpler like telnet?)
Besides, running a service on a non-standard port offers pretty good protection against a fairly common threat scenario in which a new vulnerability is discovered in a well-known service and attackers start sweeping the internet for open ports in order to exploit it.
Not that you should automatically run your services on non-standard ports; there are tradeoffs as others have mentioned. But if you've got to expose something dangerous to the internet at large (and that's an assumption that you should question frequently), then avoiding the standard ports can be a useful tool.
Since I disabled password logging on all accounts and my key is stored on an HSM I'm not too worried about it though.
Somebody that added nmap to their bot script.
It's a big quiet void out there in sixspace.
[0] https://unix.stackexchange.com/questions/105553/how-to-provi...
edit:
quick and dirty notes of my setup [1]
The SSH daemon logs when it successfully rejects an access. A successfully rejected access is inconsequential to your security. If you are using secure passwords or pubkey authentication, it will never log a successful login by an attacker. What remains then is exploitation of the SSH server ... but the SSH server doesn't have a code path that logs "I have been exploited".
I much prefer restricting port 22 to a few ip and disable passwords
http://www.cipherdyne.org/fwknop/docs/fwknop-tutorial.html#w...
It is only clear to the routers in your traceroute ... third party attackers cannot snoop on this traffic.
While port knocking is really just another layer of obscurity, obscurity works really well on scattershot/random attacks. An attacker dedicated to getting into your specific server is another matter but thankfully far more rare.
Thank you. Remember - security through only obscurity is a bad idea, but additional layers of obscurity can be very valuable.
Also remember: a typical port knock is a series of three ports that all have to be hit in a certain timeframe - for instance, 2000, 4000, 8000 - that's a big "keyspace" to brute force through at WAN packet speeds ...
But when sshd ends up pegging an entire core, and I can't login myself anymore, then fail2ban (or even whitelisting IP ranges for ssh) becomes necessary.
I mean, why would people even try.
Then there was this: https://jblevins.org/log/ssh-vulnkey (tl;dr - there was a bug in debian that caused ssh-keygen to only produce a very small number of keys rather than keys distributed across the whole keyspace). After that became public the bots were all about trying those keys for a few years after. To this day it's recommended to not used keys from that set.
Point being - the bots will try because it the tiny marginal cost is not high enough to stop the attempt.
Without it, 99.99% of the average ssh log is failed back attempts.
nftables instead of iptables
port-knocking
non-standard port
key+pass access/auth
ip whitelist
good logging
ED25519 wherever possible!!!
TalkTalk UK, jumps around a lot; YMMV.
Anyway, nftables should be nicer to administer. More pf-like.
So I made the choice to go ahead and focus on learning nft. It's been enjoyable.
The totp packet is a full hash of time + shared code so brute force is completely infeasible.
It's currently in python, it would be much better in something like go with daemon monitoring. It is very simple, and it practically eliminates attempts. If it became widely known there would probably be attempts on the udp port. An additional improvement would be randomize the listening port based on the time+hash:
I looked at that before starting and reference it in the blog post (but should probably include a direct link to the software rather than the original concept post and article that include links).
The similarities are they both use a single packet and they are both an improvement on port knocking.
This is a simpler solution (pairing UDP with TOTP). No TCP connection is required. Making it ip address specific would prevent replay attacks (although even without address specific the replay would only be valid for ~30s).
Another benefit is that a port scan will reveal no ports open (because it is a one-way UDP based protocol).
fwknop is definitely a polished solution where this is just a proof of concept for the TOTP/UDP as a shared key knock.
Thank you for the feedback!
EDIT: I updated the post to include a direct link to fwknop rather than just links to the posts/articles about it.
I don’t have logs from that server, but here are logs – just the failed attempts – of 2 months from a server that was less severely affected: https://s3.kuschku.de/public/failed_ssh [327M]
And that was despite already blocking massive areas of IP space already.
$ grep -c sshd /etc/hosts.deny
1192
$ uptime
11:58:51 up 327 days, 21:33, 1 user, load average: 0.13, 0.11, 0.09
Times have changed...
(kgraft, kpatch)
Many botnets just focus on IPv4 and there is a lot more territory to scan on IPv6. I usually couple that with moving to a non-standard port, enforcing modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc
In a classic CIA honeypot I would imagine the women are carefully matched to the target. If you are "James Bond" you get a better looking woman to try and trap you. If you are "Wallace Shawn" (in looks) you would probably figure out really quickly it's a trap if the woman looked like Angelina Jolie.
To prevent even those initial attempts, I use multiple layers of defenses such as country based blocking or blacklist based blocking. If an attacker slips through despite all this, I blacklist them manually in /etc/hosts.deny.
If there is any interest, my scripts are in https://github.com/KamarajuKusumanchi/hosts.deny/
It is fairly evident to anyone that has looked at the traffic that a honeypot gets that most of the activity is automated.
Most of it is purely botnets, some of it is automated password guessing followed by manual login. I think the amount of people manually trying passwords is insignificantly small.
No open port = no hacking attempts.
Investing a week of on-and-off studying and tinkering with openvpn is really paying off.
That's just obviously nonsense? Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
But then, there is no fundamental difference between a service rejecting unauthorized connections and a firewall rejecting unauthorized connections. If your service is already rejecting unauthorized connections, you don't gain anything by also rejecting the same connections at the firewall, and that is why "No open port = no hacking attempts" is ultimately nonsense: It doesn't change anything about the attempts, and chances are it doesn't fundamentally change anything about the rejections either.
Also, adding a VPN exposes the VPN service to the internet, which thus adds attack surface in a different place. Which might be worth it if you need remote access to otherwise vulnerable services. But the simplistic view of "rejecting connections at the firewall" == "no more hacking attempts!11" is just that: simplistic.
How about "No open port = no concern about possibly vulnerable services running on open ports"? I actually worry less about passwords and more about overflows, protocol problems and parse errors these days.
My point isn't that blocking off ports at the firewall is always pointless, but that it's usually a trade-off, and keeping a vulnerable service running behind the firewall can still be a risk, and many "hacking attempts" are just irrelevant if you follow general best security practices, so it's pointless to do anything specifically to prevent them.
https://www.blockedservers.com/
Super useful.