Endlessh-go: a Golang SSH tarpit that traps bots/scanners | Hacker News Reader

Endlessh-go: a Golang SSH tarpit that traps bots/scanners

github.com

190 pointsfastily60 comments

Golang works well for this application because it can easily cope with very large numbers of idle goroutines.

What the author may be missing is that golang also works well for bots and scanners, for exactly the same reason. Attackers' time isn't being "wasted" by this, their goroutines are just sitting idle for longer.

Though for a large amount of HTTP bots, the authors don't even bother changing the default Python User-Agent. I'd assume a large proportion of these bots still can't run concurrently.

I think it still works. You have two scenarios where the attacker is efficiently using goroutines; (1) you also use goroutines or (2) you do not. In the latter, the attack is more expensive for you.

Another detail is that an attacker with many idle connections to your host might not instantiate any new ones.

Of course, in the scenarios where the attacker is not using goroutines then you have the upper hand as well.

This isn't a real ssh server, so the "cost" to you of the attack isn't really relevant. You can choose not to run this software at all, and the additional cost to you is zero.

The original endlessh hints at this, but doesn't go further into details, and the endlessh-go's README doesn't mention it at all. Am I suppose to have endlessh run on port 22 and then have my real SSH server run on an obscure port? In none of the examples does it run on port 22. I feel like I'm missing something obvious, that the READMEs simply take for granted I know.

That's how I have it setup

Yes, check this distro https://hub.docker.com/r/linuxserver/endlessh

I don't think it matters, ssh bots will try any port that sends back the ssh banner.

The more targeted/sophisticated ones will, but there's a crapload of bots that just scan all publicly addressable IPs for port 22 and attempt to connect. If your goal is to trap as many bots as possible in the tarpit, you'll get a lot more if you run on port 22.

But they don't scan every port. I've been running my SSH server on a non-standard port for a long time, it took four years until I had the first bot with login attempts. About a year ago I changed the port and haven't seen any bots since then.

You can surround your custom port by a couple of ports on which a simple server listens for connection attempts. Any connection attempt is considered hostile and the ip will then be blacklisted in iptables. This prevents portscans from reaching your port.

Only works for sequential scans, most scanners are more targeted towards specific services.

If they're targetting SSH specifically how are they going to guess i'm running it on port 1690 and not port 22 other than by scanning up in sequence?

Different quality of locks in the ever-escalating arms race. Probably there are many many more sequential scanners out there. For the persistent actors who are doing random ordering or shuffle then you could add port-knocking for the real sshd... but then they just have to find a working client and sniff the connection requests... to which you add a TOTP step for determining which ports to use, and so on...

There is a known upper bound they could randomise the guesses from the range.

Excuse the old school metaphor - you put a lock on your door so your house is harder to break into, not to prevent anyone from breaking into your house.

Absolutely agree, when I wrote this I was thinking more of defending against the low hanging fruit - mass scanners.

Once someone has deemed you a worthwhile target and is carefully proving all ports, these more nuanced approaches become more worthwhile. Even then, a sophisticated adversary may have many unique src IPs at their disposal.

That's the theory. I have an internet facing box using ssh on a weird port with fail2ban on it just in case.

In over 10 years I've never had a single probe on that port with ssh.

Same, I went from logging thousands of attempts per day to zero per decade with a simple switch of ports.

They CAN, but they don't.

I run endlessh on the port 2222 and I configured fail2ban to redirect the source ip addresses who did X failed attempts from the dest port 22 to the dest port 2222 transparently. I use the table NAT and prerouting to achieve that, you can use ipset to match the source ip addresses.

I do something similar except send them bytes from /dev/random, providing free protocol fuzzing.

Oh nice, do you have a blog post detailing it step by step?

Isn't the point of a honeypot that it's not a real server? What guarantees are there that there won't be an exploit that allows escaping the honeypot into the real data? Personally, I do not believe anything is 100% secure. So inviting the vampire into your facade home, and then getting upset when the vampire sees the charade and walks into your real home is just one of those "well of course that happened" situations.

if you use port knocking, the first hit on your honeypot, can be the trigger to lockdown or redirect, a lot of other ports to somewhere away from your actual.

Following the SSH hardening guide stops 99% of bots and scanners because they can't negotiate a cipher using whatever ancient ones their SSH implementation is set up to use.

This, and a handful of simple firewall rules in the raw table can block about 90%+ of that remaining 1% just looking at the spoofable banner that none of the bots seem to spoof I assume due to being lazy like me.

In the raw table:

    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-libssh" --algo bm --from 10 --to 60 -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-Go" --algo bm --from 10 --to 60 -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-JSCH" --algo bm --from 10 --to 60 -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "SSH-2.0-Gany" --algo bm --from 10 --to 60 -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "ZGrab" --algo bm --from 10 --to 60 -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "MGLNDD" --algo bm --from 10 --to 60 -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [my server ip] -m string --string "amiko" --algo bm --from 10 --to 60 -j DROP

Adding the server IP minimizes risks of also blocking outbound connections as raw is stateless

I rarely do this any more given they rotate through so many LTE IP's. Instead I get the bot operators to block me by leaving SSH on port 22 and then giving them a really long VersionAdendum that seems to get the bots feeling broken, sticky and confused. There are far fewer SSH bot operators than it appears. They will still show up in the logs but that can be filtered out using drop patterns in rsyslog.

    VersionAddendum "  just put in a really long sentence in sshd_config that is at least 320 characters or more"

Try it out on a test box that you have console access to just in case your client is old enough to choke on it. Optionally use offensive words for the bots that log things to public websites. Only do this on your hobby nodes, not corporate owned nodes unless legal is cool with it, in writing.

Why use PREROUTING chain? You could have achieved same with INPUT chain without specification of ingress interface and server IP address.

Anything I explicitly drop I do so in the raw table to keep them out of the state table. The state table is more CPU expensive especially at high packet rates and runs the risk of depleting the default state table limits especially for anything that now has a broken state on purpose like these poor lil bots. Since I brought it up, here is how to increase the state table limits.

Create /etc/modprobe.d/nf_conntrack.conf

    cat /etc/modprobe.d/nf_conntrack.conf 
    options nf_conntrack expect_hashsize=256400 hashsize=256400

And then in /etc/sysctl.conf:

    # from /etc/sysctl.conf: increase state table limits.
    # Requires 1/4 mem to hash table plus 400 overhead because I am the cargo culting king:
    # cat /etc/modprobe.d/nf_conntrack.conf
    # options nf_conntrack expect_hashsize=256400 hashsize=256400
    net.nf_conntrack_max = 1024000

Should people use default state table memory allocations on a busy node, everyone can be locked out of it regardless of how many TB of RAM are free. The node can appear "down".

include EICAR.TXT ;)

I don't know if this is still the case, but -m string used to be resource intensive, because it has to parse each packet for the string before passing it on to other rules.

It can be. This this case however it is limited to eth0, tcp, port 22. If any of those don't match there will be no parsing and thus no impact. Another mitigating factor is that we are only looking at specific byte regions of the packet so parsing is minimized. On busy SFTP servers I would probably avoid using such rules if CPU load is becoming a problem. For most people this will not even register in htop or vmstat. There are also ways to use this string check in combination with ipset and/or xt_recent to minimize the times we see a packet from a bot. Here is an example using an IPSet called "bots" that we drop early on in the raw table and also use in the filter outbound rules to reset openssh trying to respond the first time we see the bad string so we close the socket earlier.

In a startup / init script / systemd unit file:

    # IPv4
    ipset flush bots 2>/dev/null
    ipset create bots hash:ip hashsize 2048 maxelem 65536 timeout 604800 netmask 24 2>/dev/null

    # IPv6
    ipset flush bots6 2>/dev/null
    ipset create bots6 hash:ip hashsize 2048 maxelem 65536 timeout 604800 netmask 64 family inet6 2>/dev/null

In this example I am using a bigger netmask much in the way name servers rrl rate limit.

In the raw table, drop bots we saw for a week:

    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22:80 -d [server ip] -m set --match-set bots src,dst -j DROP
    -A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -d [server ip] -m string --string "SSH-2.0-libssh" --algo bm --from 10 --to 60 -j SET --add-set bots src --exist --timeout 604800

In the filter table outbound rules:

    -A OUTPUT -o eth0 -p tcp -m tcp --sport 22 -m set --match-set bots dst -j REJECT --reject-with tcp-reset

This should only be performed on servers that one has console / out-of-band access to, after exhaustive testing.

But the bots can easily detect these, cant they? As long as there is a timeout on socket read, this shouldn't waste that much of the scanners time.

Or am I misunderstanding this?

Yes a bot can, and sophisticated bots do.

At the same time it's much easier to write code that just died the bare minimum. Imagine you're a bot herder, if your bot net consists of stolen CPU cycles what difference does it make if your bots are slowed down. It doesn't cost you money.

> if your bot net consists of stolen CPU cycles what difference does it make if your bots are slowed down. It doesn't cost you money.

This is wrong. It does cost you money - either directly, because you paid money to use someone else's botnet, or as an opportunity cost, in that you can't use your bots on as many targets.

Endlessh periodically sends data so the read timeout won't trigger. Specifically, it draws out the crypto negotiation stage indefinitely by exploiting a feature of the SSH protocol.

(Of course, the bot author could detect that behaviour too.)

There's more info from the author of Endlessh: https://nullprogram.com/blog/2019/03/22/

You're understanding perfectly. The way this works is that it sends a slow drip of junk before the SSH version banner string. A scanner running at any real scale is going to have an overall timeout beyond which it doesn't bother waiting any longer for the banner string.

This is going to very slightly irritate some of the extremely low-level actors. Is setting up a tool to do that a good use of time?

If you want to effectively deter attackers using a sand-trap approach, you need to find some kind of task with asymmetric cost in your favour. This isn't that.

What is a good example of assymetrical cost in your favour?

CAPTCHAs are an example. Although I am not sure if they're a good example.

You could probably achieve better here by providing fake weak credentials and then getting an actual human to connect and check out the honeypot on as many IPs as possible.

Funny but my first thought wasn't wasting their time at all, that's easily fixed with a few code adjustments on their client end. My thought was to harvest their IPs and publish them in blocklists.

> My thought was to harvest their IPs and publish them in blocklists.

Please do, it would mean good karma.

abuseipdb.com

I think you could employ the same tactics that advanced fuzzers do with these tarpits: then mutate the responses randomly, to try get "new" responses from the attackers, instead of new coverage in the code as in the fuzzer. Unless they are using static scripts, which would be boring.

I have understood that most attacks are super-simple sort of, so probably not much to learn there. But an interesting project!

For other usecases there is an ipfilter target TARPIT, that does a similar thing on the TCP level.

But why? Just hide your ssh port. And port knock in. Or put it @ tor/wg/whatever.

Does it matter, though? You can easily scan out the correct SSH port.

Scanning all 65k ports takes time. Those aren't targeted attacks, just bots connecting to every 22 ports they can find

Depends on how well programed the bot is I guess. Personaly I use the encrypted packet port knocking package fwknop on my home server to hide the ssh port until I need it.

The point of this isn't to hide your actual SSH service, but to tie up resources for those who are somewhat blindly scanning/connecting to any open SSH port.

You can setup a VPN (or head/tailscale) and confine your "real" sshd there, and leave one of these tarpits in the open for fun and profit.

> Unfortunately the wonderful original C implementation of endlessh only provides text based log, but I do not like the solution that writes extra scripts to parse the log outputs, then exports the results to a dashboard, because it would introduce extra layers in my current setup and it would depend on the format of the text log file rather than some structured data. Thus I create this golang implementation of endlessh to export Prometheus metrics and a Grafana dashboard to visualize them.

" I didn't like the logging, so I re-implemented the entire thing."

I'm not mocking, I just see this often (and have done it myself!). It's interesting the things we do to get around the little things we don't like.

The thing about coders is that they're often creators at heart.

The stated reason is likely only the excuse they told themselves to justify the project. But the real reason was likely that they wanted to create something, and this was a good justification.

Might just be me projecting though, because I do that all the time

I've stopped lying to myself, I'm making cool stuff just for the sake of it. It's like playing a video game, it doesn't have to be productive if I enjoy it.

> " I didn't like the logging, so I re-implemented the entire thing."

And did that in the "language of the week" :)

The stuff that was reinvented in eg. ruby a few years ago is now reinvented in go and rust.

People build in languages they are comfortable with. Golang is fairly easy to ship with and the performance is fine for something like this. Seems like a good tool for the job

In fact, Go is one of the better tools for the job given its ability to easily spin up network connections.

I was more pointing out the fact that people reinvent existing stuff in a new language, then a few years go by, the 'existing stuff' is still 'existing', the 'new language reinvention' is dead, and someone new then reinvents the 'existing stuff' in some new language :)

eg: https://news.ycombinator.com/item?id=19276751

https://github.com/remacs/remacs

last commit, 3 years ago.

> It's interesting the things we do to get around the little things we don't like.

Yeah and perhaps pick up valuable skills, that might help us down the road in ways that are hard to quantify.

Reply on news.ycombinator.com