Learn from your attackers – SSH HoneyPot
robertputt.co.uk
robertputt.co.uk
But, we also had every customer on a VLAN, limited to only being able to send traffic from their IPs, and also blocking incoming and outgoing bogon traffic.
Years ago I attended a presentation by Evi Nemeth (RIP) related to CAIDA and one thing they found in auditing "backbone" traffic was that some huge percentage of it was bogon traffic (I don't recall the exact number, but lets say 10% +/- 6%). Nobody wanted to filter that traffic because the pipes were less expensive than the routers to handle filtering packets at high pps rates.
Yes, there could have been compromises that went entirely undetected for years (we had a really high retention, so most customers stayed with us for 5+ years), so we had a good window to detect issues.
Probably the next biggest notification of compromise was alerts about spam on our network. Mostly that was an e-mail account compromise rather than system level. But there are a ton of false positives on spam alerts, particularly AOL alerts were almost always about legitimately sent e-mails.
Situation is better nowadays, modern routers like Juniper's Trio-based MX platform do things like bogon filtering and reverse patch checks at line rate without performance impact.
It's secure, they can't break out of the jail.
It's rate limited to prevent them causing much damage to anyone.
It's easy to observe every thing they type and do in the jail from the host.
* Download a large file (e.g. XP SP3) from Microsofts servers to test bandwidth * Download IP scanners and trying to run them (in order to find new hosts to attack) * Download botnet code that connects to some C2 server
Tools they download were often downloaded as source, which they then tried to compile in order to run it.
Searching for e.g. 'youtube kippo' will get you videos of honeypot sessions.
security.bsd.see_other_gids
security.bsd.see_other_uids
The above are the two sysctl's you want to enable.
A simple ps will show you all the shells that are in a jail.
Tue 19 7:19PM priyanka.setecastronomy ~> ps -x
PID TT STAT TIME COMMAND
30308 - S 0:00.05 sshd: oracle@pts/0 (sshd)
Now you have the tty he/she is on (pts/0). There are many ways to get that including ps so use whatever method you prefer to get the tty.
Fire up watch to snoop on the tty.
Tue 19 7:48PM priyanka.setecastronomy ~> doas watch pts/0
And boom, you can watch the ankle biter do his/her thing.
You can do this on the host as described, because that is how pam_jail works. But if you prefer for some reason to make an actual jail with VNET etc and do this simply forward ssh on the host to the jail's IP. If you do rate limiting you will already be configuring pf or ipfw anyway so an extra line to forward 22 to the jail host is not big deal.
Yeah this is a crappy write-up but it's spur of the moment and in a HN reply post so it's worth the price paid.
I'm happy to answer questions or help you if you get stuck anyway I can just message me. It's pretty simple to do.
If this is in a corporate setting I would consult with legal before doing this. The world is a whacky place these days, and if the ankle biter does something evil from your honeypot jail you may be liable.
I would sit there and watch that.
Oh, look honey, this one's trying to use GNU options! Here's one who thinks he's on Ubuntu! Wow, I didn't know VIM could do _that_!
Love those references. If you have not seen it yet go watch Sneakers (1992). I am getting old - I remember when it was fresh :-)
To make it even more interesting: one can actually see which bot is trying by the way they fail to handshake.
I like a litte noice on my ports so leave ssh on its default port, accept authentication only with a certificate and be done. Works like a charm. And yes it makes my sshd stand out like a sour thumb since 99.9% (guestimation) of you folks don't tighten up your ssh configs :)
I'm withdrawing the question per the accepted answer below.
-
original comment:
>it's just ridiculous and important to use non password authentication with ssh
Does this follow from the assumption (that you did not state) "since I won't bother to create a high-entropy password"?
Granted its a fairly safe assumption if you're not trying too hard but SSH supports passwords of arbitrary length so I'm curious if there's any other reason, besides not taking the time to create a high-entropy password.
-----
Accepted answer below (esp "Because your users cannot be trusted. I run servers which coworkers, all developers, get access to.")
/etc/security/pwquality.conf
if you use keys, this doesn't happen.
Edit: I didn't mean changing the port is a bad thing. You can and should do it, and it will help a bit - just that it shouldn't be the only thing to rely on for your SSH security.
It's not the only solution but something that can be used to at least drastically reduce the amount of noise in logs.
> I've found that it's more efficient just to whitelist the IPs or subnets you might need and denying the rest.
That does sound more efficient but what if someone connecting doesn't have a static IP?
Whitelist your ISPs subnet then, at the very least - there's much lower probability of an attacker coming from just your ISPs, and, of course, use other measures too - I didn't mean that to be the only solution for the problem.
I'd argue they are effectively the same thing with the same auth methods available for both.
Why do you consider VPN to be a better protocol than SSH for security/authentication?
Attackers who are not specifically targeting the company are unlikely to bother or to have the skills necessary. It's too much hassle and attackers scanning around for a random victim don't even see that the ports are open since they're behind a firewall.
So, it's not about this protocol vs that protocol for authentication, there certainly shouldn't be open access just because you got past the VPN auth, it's about layering your security to shrink the surface area and make it simpler to identify the entry points.
I guess I never considered the need to SSH directly to machines. I completely agree security is defense in depth through layers.
I brought up the question since I was just at a client site that required VPN to ssh to their single bastion (jumpbox) ssh host, which then you used to further ssh to production machines. The VPN layer in that architecture seems rather silly - just lock down your bastion server (SSH) in the same way!
No, I apologise if I wasn't clear, the bastion server is accessible only after you are either also within the office or logged in via the VPN. It's an additional hoop to jump through, not a replacement. The point being, the bastion server is locked down in addition to the requirement to be logged into the VPN or being physically located at the office. IP locking is not a replacement, it's a sensible addition. I think perhaps this is why you think it's silly; you're misunderstanding that it's not a different layer it's an additional one.
Security is about layers. Nothing is foolproof. It's about implementing layers of controls to reduce your attack surface to an acceptable level, with the trade-off that many controls increase the complexity of your setup or compromises the convenience for your users.
For example, for SSH, this probably includes
* changing the default port
* enforcing SSH key authentication
* enforcing passwords on SSH keys
* implementing fail2ban
* installing jump hosts for internal machines
* implementing a VPN rather than external facing hosts (and with that comes all the additional layers for the VPN)
* etc...
That cannot be enforced by the server because the key decryption occurs client-side. An alternative is to use Two Factor Authentication.
And 2 or even 3 factor should maybe be on the list (key+pw, key+totp, key+pw+totp).
For keys, it's in theory possible to ease management with using ssh certificates and a CA - anyone know of a convenient way to manage totp secrets across multiple servers and users?
This might help in forensic investigation afterwards. Less crap to wade through.
Let's face it, the sort of person who moves SSH to a different port probably is also the sort to disable password logins. So the effectiveness of searching for their SSH daemon is probably pretty low.
Every remote machine gets it's own keypair as well and links between machines also get their own key-pairs (key handling is PITA, I've yet to find a method I really like), Keys for work stuff are back up on multiple LUKS memory sticks (two in separate safes, two offsite).
I'm fortunate in that work machines are in the pet category not cattle category so we don't need to change keys that frequently (though I will rotate them out yearly anyway).
It might be better not to disclose this level of detail about your professional security practices, especially when there is a trail of breadcrumbs in the HN profile on your account..
The default ip(6)tables rule: drop all packets
For example to access my least secured public facing server you need to do the following: 17 port port-knocking to enable crypto port knocking Which enables ssh Which requires a 16K RSA key and TOTP The only accounts that can be logged in have 18 character randomly generated usernames and 255 character passwords. The login accounts have no access to administrative functions. So you'll need to su to an administrative account and guess the 255 character password or have a zero day exploit that can bypass SELinux and cryptographically signed white-listing of binaries.
The first one don't attack much of anything, there is the occasional spider that tries to index your website (we had data worth scraping). The later can be banned by entire AS without issues.
1% is a pretty low threshold. There are some v4 networks out there that are complete garbage.. If I was going to do something like that I'd start at closer to 90%.. 230 hosts on a /24.
$ select subnet, count(distinct(cidr)) as unique_sources from (select set_masklen(cidr,16) as subnet, cidr from stuff where why like 'SSH%' and added > '2017-08-01') as foo group by subnet order by unique_sources desc limit 20;
subnet | unique_sources
----------------+----------------
181.211.0.0/16 | 11688
190.214.0.0/16 | 8486
31.162.0.0/16 | 8454
181.196.0.0/16 | 7994
181.113.0.0/16 | 7892
188.16.0.0/16 | 7294
94.51.0.0/16 | 6384
188.19.0.0/16 | 6077
31.163.0.0/16 | 5905
178.47.0.0/16 | 5788
201.178.0.0/16 | 5620
190.48.0.0/16 | 5179
188.17.0.0/16 | 4893
201.179.0.0/16 | 4812
188.18.0.0/16 | 4266
186.178.0.0/16 | 4208
5.141.0.0/16 | 4203
186.129.0.0/16 | 3858
181.112.0.0/16 | 3836
190.174.0.0/16 | 3836
90% seems really high... you'd really wait until 230 of 255 possible hosts have attempted a breakin before deciding they were on a network too dangerous to preserve your accessibility from? Are there a lot of networks where 90% of the boxes are launching attacks, but 10% have legitimate need to connect to your personal home machine?
The bulk of the abuse comes from Russian and Chinese networks, with Brazil working hard to catch up. Those can be pretty well perma-banned without consequence. Under "folks that should be fined into oblivion", there's cogentco.com, quadranet.com, colocrossing.com, level3.com, vpls.com, softlayer.com, hostingsolutionsinternational.com, datashack.net, singlehop.com, actonsoftware.com, and a whole raft of others.
The secret sauce is a carefully tuned ban decay function that scales up with the number of abuses. Occasional hits get a network banned for an hour or less, subsequent hits get them banned a little longer, and then there's a fine line where the function goes exponential, all the way up to 6-month-plus ban periods for really big nuisances.
Otherwise it's just fail2ban on some logs and some home-brew code that does whois lookups on nuisance IPs and some data mining on the whois response.
Of course the Chinese government are aware of existing circumvention, but they still allowed them because they want to capture data. They let these circumvention to exist even though they have the brightest engineers in China working for them. I vaguely remember a talk from RealCrypto a number of years ago, someone presented how one could try to escape GFC by sending data over different protocols and then resemble by the recipient. It's an ingenious circumvention that is quite hard to decipher, until someone recognizes a traffic pattern, then Chinese government knows.
As we are really only interesting in our state, I don't think twice about banning international IPs and they are mostly from large datacentres.
What do you mean these should be fined? Level3 is a transit provider for massive portions of traffic on the Internet and Softlayer has some hundred thousand servers.
Wasted time for precisely zero benefit.
There are lots of hobbyists that'll tell you otherwise, but in reality you should be using key auth and worrying about better things.
There's a ton of useful open-source tooling one can play with in the space. Here's a list:
- https://www.smokescreen.io/practical-honeypots-a-list-of-ope...
Most of the projects are also small enough that you can easily contribute back to them especially in terms of reducing the ability to fingerprint the honeypots.
Somewhat timely, The Grugq just wrote up a nice article on Counterintelligence for Cyber Defence:
- https://medium.com/@thegrugq/counterintelligence-for-cyber-d...
I think it's just a matter of numbers. The IP search space is already pretty large with IPv4 on just a single port. With IPv6 it'll be more or less impossible for attackers to make any substantial progress by just searching for IP addresses to attack, let alone an IPv6 address on a non-standard port.
After reading this I'm likely going to be upgrading my technique. SSH on IPv6 only (eventually shutting down IPv4 altogether, can't wait), listening on a non-standard port, limit 1 connection per IP.
Usually, you want one full-height terminal and two about half the height of your screen.
I do most of my work on our university cluster over SSH. I can close/shutdown my laptop at any time (or have my connection drop) and as soon as I SSH back into the server all my editors, shells, running jobs, etc. are exactly how I left them.
iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 180 --hitcount 4 --rttl --name SSH --mask 255.255.255.255 --rsource -j DROPWith a nonstandard port, bots mostly don't even bother to scan you.
It just really is a part of your password (albeit one that can be guessed easily by a more dedicated attacker, just like the username).
The added entropy is a bit harder to calculate, but unless your password has 16 bits of entropy¹, it will round pretty well to zero.
1 - Why TF would you run a ssh server that accepts passwords anyway? But nearly everybody I have seen that thinks changing the port is a good form of protection also sees no problem on accepting them.
I ask because systemd may not be writing the failed login attempts to the log file fail2ban monitors, which means fail2ban may not be responding to attacks. I don't recall exactly how I resolved this, but I think I had to set "backend = systemd" in my jail.local under the [DEFAULT] section header.
Yes, that's what I do. If you have SSH login via password disabled (you should) then just temporarily move your private key(s) out of your ~/.ssh directory and try logging in a few times. "ssh user@machine && ssh user@machine && ssh user@machine" will do it if you have a ban set for three failed attempts. If it's working, your target machine should no longer respond at all, and connections will time out. If it's a web server the site should be unreachable with your web browser until after the bantime passes.
btmp begins Fri Sep 1 06:25:05 2017