Endlessh: An SSH Tarpit | Hacker News Reader

Endlessh: An SSH Tarpit

nullprogram.com

579 pointsstargrave112 comments

Neat little project to needle some of the botters a bit (though I assume they'll all evolve to recognize this fairly quickly). I'd echo tyingq's comment below though:

Any tarpit has the potential to piss someone off. I'd run it on a sacrificial server with no obvious way to tie back to who is running it.

Yeah, just speculating but at least from what I've seen in the past if you successfully tarpit some script kiddie and they notice the IP their scan got stuck on there is some potential to move from "one of a billion random lowest common denominator bulk scan targets" to "paid attention to specifically some minimal amount", which is a genuinely different scenario. Even if all that amounts to is a relatively low volume revenge DDOS pointed at you for a bit it's still more of a disruption then if the auto scan had just moved on without seeing anything of note in the first place. This looks like fun on some systems, but on anything real I'm inclined to just stick to cutting down on log spam via single packet auth or a port knocker or the like. The old outrun-the-hiker-not-the-bear aphorism fits a lot of cases, just something to keep in mind before implementing something like this if you aren't directly experimenting with more active reactions.

Conversely as a research project I'm now actually curious what sort of extra attention even something like this could attract. Maybe these days everyone would just adapt and move on instead and the above is all obsolete?

Good advice, being ready for a revenge DDOS, but I suspect most of these SSH hunters are busy business people who have a quota of zombie hosts to fill for whatever mining or spamming they're up to. DDOSing doesn't pay like those things do.

Exactly.. figuring out why one host out of 500000 you probed got hung up is generally not a wise use of a bot operators time. Now if enough of these tar pits start showing up you can be sure they will quick patch a time limit to defeat stalls like this.

Even sophisticated/organized hackers have free time, and if someone "messed with them" and caused them to need to push a hotfix at 6pm on a Friday because their cluster got stuck on your tarpit, you can bet they'll give the tarpit operator some of their attention. Woe to you if they happen to find something that's not patched.

Turn it around. How would hackers adapt if nearly every host responded on ssh the same way? Maybe it’d be behavior worthy of being in openssh or elsewhere. Don’t make it cheap to cheat. This might not be the holy grail, but it’s a good avenue.

Seems to me like they'd just implement a timeout on waiting for the identification string. I don't know the SSH protocol that well, but from the article it sounds like it's easy to bypass this tarpit if you don't strictly follow the SSH standard. I doubt a timeout would have much impact when connecting to legitimate hosts since they probably respond quickly with their identification, so the fix is easy.

Realistically if you do a massive scan you have to handle all kinds of behaviors and this would be a debugging point basically ("oh my script is stuck, let's add a timeout").

The thing is most people will not see the trap as malicious but as a "misconfiguration" on their end or the other person's end (network or ssh in this case).

I see your point but then I never heard of honeypots getting some kind of "revenge" for instance.

I like this! It's tiny, simple, yet effective.

    2019-03-22T19:54:06.303Z ACCEPT host=::ffff:196.52.43.xx port=50327 fd=4 n=1/4096                                        
    2019-03-22T19:54:38.838Z CLOSE host=::ffff:196.52.43.xx port=50327 fd=4 time=32.535 bytes=199

    2019-03-22T19:57:12.008Z ACCEPT host=::ffff:141.98.81.xx port=53646 fd=4 n=1/4096                                        
    2019-03-22T19:57:21.118Z CLOSE host=::ffff:141.98.81.xx port=53646 fd=4 time=9.110 bytes=30

The 141 bot never stays longer than 5 seconds. I'm guessing they are using 'expect' and waiting 5 seconds for a particular string.

The only thing missing is setting some glibc hardening options

    RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH FORTIFY Fortified Fortifiable  FILE
    Partial RELRO   No canary found   NX enabled    No PIE          No RPATH   No RUNPATH   No 0  8 ./endlessh

I modified one line in the Makefile

    CFLAGS  = -std=c99 -Wall -Wextra -Wno-missing-field-initializers -D_FORTIFY_SOURCE=2 -O2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -ftrapv -s -g -Wl,-z,relro,-z,now -Wl,-z,noexecstack -pipe -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection --param ssp-buffer-size=4 -fPIE -pie -m64 -mtune=generic

It's better now

    RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH FORTIFY Fortified Fortifiable  FILE
    Full RELRO      Canary found      NX enabled    PIE enabled     No RPATH   No RUNPATH   Yes 3  4 ./endlessh

You should submit a pull request!

The thought crossed my mind, but I am lazy and the proper change would be to write a configure script that checks for the availability of the various hardening options.

There are decent examples in the Tor source code.

Thanks for introducing me to `checksec`!

My pleasure!

Suggestion for Debian: if you get error: unrecognized command line option ‘-fstack-clash-protection’; did you mean ‘-fstack-protector’, do:

echo "deb http://ftp.us.debian.org/debian/ testing main non-free contrib deb-src http://ftp.us.debian.org/debian/ testing main non-free contrib" > /etc/apt/source.list.d/debian-testing.list

apt-get update

apt-get install gcc-8/testing

The Makefile can then contain the above line:

    CFLAGS  = -std=c99 -Wall -Wextra -Wno-missing-field-initializers -D_FORTIFY_SOURCE=2 -O2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -ftrapv -s -g -Wl,-z,relro,-z,now -Wl,-z,noexecstack -pipe -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection --param ssp-buffer-size=4 -fPIE -pie -m64 -mtune=generic

and:

CC = gcc-8

Replace -mtune=generic by your CPU, cf https://wiki.gentoo.org/wiki/Safe_CFLAGS

Do you realize that the thing does not process any attacker controlled input and thus all the hardening is completely pointless exercise?

The number of times in my life that I have seen "this should never happen" scroll by on STDOUT suggests that it's time well spent.

If you cannot trust that 725 line program without any mention of recv() nor read() will never call recv() or read() on network socket then you have much larger reliability (not security, because whether something that is broken to the point of being unusable is secure or not is completely irrelevant) issue in your system that is not going to go away by applying random hardening options to said program.

If it opens a socket, it gets hardening. It's just a best practice I follow. Code can change. Subtle changes can lead to gaps, so I wrap that rascal regardless.

I've been running this for a couple days now. In summary, about 85% of the bots give up in 5 to 30 seconds. The remainder of them will remain stuck anywhere from 700 to 9000 seconds. It is at least interesting to see how bots have evolved.

This is the Internet equivalent of keeping phone scammers on the line by not just refusing to hang up, but actively interacting with them until they hang up (search YouTube for "tech support scammer trolling" for some entertainment...)

That said, I'm a little confused even after reading the linked RFC paragraph; it claims "The primary use of this feature is to allow TCP-wrappers to display an error message before disconnecting." but I'm not familiar with that term and there are no other occurrences of the word "wrapper" in that document.

Firefox and Chrome will spin on that server for hours before giving up.

For a web browser, this behaviour makes a lot of sense: timeouts are usually based on "I've waited for X and haven't received anything", not "this connection is like collecting water from a dripping tap". As long as there is a slow but steady stream of content coming in, you wouldn't want to abort until the user gives up or an internal limit on the data collected is hit (which is probably what happened with the author's tests --- a quick search doesn't yield the maximum size of the response headers browsers will accept, but servers usually have a configurable limit of a few KB at most for the request headers alone.)

TCP Wrappers was a popular access control mechanism in earlier decades: https://en.wikipedia.org/wiki/TCP_Wrappers

It does a number of things in userspace (with a simple minimalist code base) that people might nowadays more often do in a firewall.

> actively interacting with them until they hang up

The best stuff is automated. Lenny has been at it for years now.

https://www.reddit.com/r/itslenny/comments/aylq20/lenny_qual...

I opened up port 22 once so that it connected (without crypto) and issued the phrase,

YOU ARE STANDING AT THE END OF A ROAD BEFORE A SMALL BRICK BUILDING. AROUND YOU IS A FOREST. A SMALL STREAM FLOWS OUT OF THE BUILDING AND DOWN A GULLY.

and if they typed in something it would continue to play the game. After hundreds of thousands of automated SSH worm probes over several months, an actual human did connect and play the game for a few minutes! But even this triumph was short lived, they did not even play it to the end to see if something interesting might be unlocked (it would have been an invitation to contact my email address).

I haven't used it in years, but Tom Liston's "Labrea" tarpit is an interesting take on this concept: https://github.com/Hirato/LaBrea

It listens in your unused IP space and both tar-pits scanners and creates actionable intelligence about scans against your hosts.

I love the fake arp responses, that's a brilliant way so you can run this on any machine (doesn't have to be a router) and not worry about managing new hosts (no black/whitelist to manage of which IPs are unused).

I'm definitely not going to try this because it sounds like an easy way to get in trouble, but would this work on a host like DO/linode/vultr, etc? Again, this seems really easy to detect

Yes be super careful with Labrea on VPS providers and dedicated server providers. They tend to get upset if you are trying to pick up IP's. In most cases it won't even work on KVM hosts.

I've used Labrea on unused address space to make ARIN/RIPE happy back in the day and that was harmless, as entire /16's and /17's were unused. They used to nag about nothing in the /16's being pingable, so Labrea made it all 100% pingable

Why hold onto addresses you're not using? This is against the allocation requirements and lead to the IPv4 exhaustion we have now.

If you have an ARIN allocation, there is no reason you would ever want to decrease the amount of addresses you own. You own them.

Whether you own them or not is subject to a lot of debate and not clear cut. If you owned them, why would you need to fool ARIN into believing they are being utilised?

The agreement you make to get an allocation requires a certain level of utilization depending on the size of the block to prevent squatting and exhaustion of resources. This is a good thing, and if it was actually respected we wouldn't be in a mad rush to migrate to IPv6.

"Mad rush" - heh. I'd hate to see how long something rolled out at a leisurely pace would take!

> IPv6 is intended to replace IPv4. In December 1998, IPv6 became a Draft Standard for the IETF, who subsequently ratified it as an Internet Standard on 14 July 2017

it only took 20 years to standardize after all. we're seriously rushing here!

Oh I was thinking home network or some small business. I'm not yet used to this cloud thing, still hosting my email and website at home. For a network that is not yours, like at a VPS provider, I would totally not try this. They would not be amused.

If you don't want to waste resources and run a separate service, you can do a very similar thing with just iptables. Look for the TARPIT target.

> Captures and holds incoming TCP connections using no local per-connection resources. Connections are accepted, but immediately switched to the persist state (0 byte window), in which the remote side stops sending data and asks to continue every 60-240 seconds. Attempts to close the connection are ignored, forcing the remote side to time out the connection in 12-24 minutes.

I'm guessing this may not work, since OP's app sends every 10 seconds and seems to imply that there is some timeout associated with receiving data from the server, but not necessarily a timeout associated with receiving the correct 'SSH-' string needed to move on.

This is awesome. Like the author, I run my real SSH server on a nonstandard port. Unlike the author, I run nothing on the standard SSH port.

I think that I'll start running this there.

On a different note, I love this other tool he wrote: Enchive

https://nullprogram.com/blog/2017/03/12/

Nice ! Reminds me of this: https://bitcointalk.org/index.php?topic=196378.0 using bitcoin keys for encryption in addition to handling money. https://github.com/jackjack-jj/jeeq

Wouldn’t cycling over the lines of an ASCII middle finger be more efficient and equally as useful as generating random strings? The middle finger would fit in a few cache lines at most.

Massively async ssh scanners won't care. They'll time out and move on without being held up at all.

I can confirm that most ssh scanning attempts/scanners I've seen implement timeouts and will usually move on after 30 seconds or so.

I wonder if you could make the attacker crunch out some shah-256 and you can profit by mining Bitcoin.

Not bitcoin but forcing a proof-of-work scheme has been proposed as an anti-DoS measure before: http://www.csc.kth.se/utbildning/kth/kurser/DD143X/dkand12/G...

The original hashcash algorithm was meant to quench email spam which is not completely unrelated.

But in this case I think the parent was proposing tricking the attacker into computing a PoW without realizing it, which sounds pretty tricky.

Just need a protocol that allows for server initiated key renegotiation with PBKDF. Then you trick the client into repeatedly calculating some new key.

You can no longer profit mining Bitcoin with a CPU or GPU. ASICs are needed.

The idea of this sort of scheme is that it's perfectly profitable if you aren't paying for the CPU and the power it uses.

Is a poll necessary?

Simple timer to count to 10. Go over all open sockets, drain incoming, pump junk. Handle accept. Repeat.

Would be nice if it logged something fail2ban could act on. So it could ban the IP-address - after bugging the bot for a couple hours.

Neat little tool! Since switching to Teleport I haven't had OpenSSH listening publicly for a while, just reverse proxies. Still, I'll prob spin up a small server for this to observe the patterns and IP locations.

Also, don't think that posting this blog article on March 22 didn't go unnoticed :)

What is significant about March 22?

SSH's default port is 22.

SSH brute-force attacks still pose a significant risk. See our related paper on SSH auditing driven by attacker's attempts here https://pmcao.github.io/caudit/

I wrote one of these for HTTP: https://github.com/carlmjohnson/heffalump

I found that disabling password authentication makes ssh probing mostly stop.

Attackers that are smart enough to detect and avoid pubkey only hosts will probably easily adapt to this program.

Perhaps this could be useful if you ran it on a vast number of ports. Just running it on the one doesn't seem to make an attacker's life much more difficult.

Running it on just 22 should be pretty effective on it's own. Large scale port scanning is semi-difficult, as you'll get banned really fast, even if you're just scanning an uncommon port. To pull it off, you'll either need your host to be okay with the abuse complaints (that are usually automated) or have a botnet/proxy list.

However even then there's 65535 different ports (not to mention almost 4.3 billion ipv4 addresses), and you'll likely can firewalled off pretty fast. So instead of scanning 65535 ports, scanning just 22 usually works anyway, as few people change their ssh ports (and those that do are more likely to have other security measures in place)

That would be easy to do with an iptables DNAT rule. You can take a large set of ports and forward them to this daemon.

    iptables -t nat -I PREROUTING -m tcp -p tcp --dport 1:79 -j DNAT --to-destination x.x.x.x:2222

    iptables -t nat -I PREROUTING -m tcp -p tcp --dport 81:442 -j DNAT --to-destination x.x.x.x:2222

...etc

Just remember to open those same ports in the INPUT rules. If you get DDoS, then also create NOTRACK target rules in the raw table to avoid hitting the conntrack table.

You truly sound like a bender. Is there a quick start to iptables somewhere? Is iptables the defacto network tool?

And can one prevent ddos using iptables?

iptables is a firewall/NAT interface. nftables is newer and easier to learn, at least to me. You might want to get into that instead of iptables.

> And can one prevent ddos using iptables?

No, but you can minimize the impact or effectiveness of the attack. This is a long topic however. Probably better suited to a blog post or a youtube video.

Set this up on a server last night. In 20 hours I've got 75 connections. Took a capture of trying to connect to it from my laptop:

https://www.cloudshark.org/captures/b64150b66908

IP addresses have been change to protect the innocent. This is from macOS but definitely seeing some different behavior from other clients.

This is great but what happens when clients start watching out for strings they don't expect? Seems like only a matter of time.

Awesome! Love reading hacks like these.

Assuming you run a legit SSH server on port A and an SSH tarpit on port B. What hinders an attacker to connect to port A an B at the same time? What is the advantage having an attacker connected for weeks on port B?

They will usually try port 22 for SSH, only. So, if tarpit is on port 22 and real SSH is on another - mission accomplished: you're making them (at least one of their threads) wait.

nothing, except the attacker doesn't know what A is. 22 is the default for ssh.

I can recommend setting up a server and do extensive logging. Then watch hackers as they attempt to break in. I had a server hacked last week and today i read the logs to figure out how they did. From first discovery, trying to figure out what system they're on, to gaining a shell, and figure out how to execute stuff, gaining network access. It was better then watching an actual movie. And a humbling experience. But also entertaining, watching the struggle against my hardening and even stumbled on a bug, feeling his/her frustration. And the wow feeling when they finally got a shell.

Postmortem on your blog?

What a wonderful opportunity to rickroll ... to prolong detection time rot13 it or walk through all the ciphers you can get away with.

cause we are never going to give you up ;)

very interesting, and useful (including newbies like me who do not appreciate the complexities and dangers of having internet-accessible IP address).

I wonder if security-oriented OSs like openBSD could offer something like this out of the box. So that an admin can just say enable_tarpits='yes', and it would automatically enable tarpits like this for all the currently installed packages (with some default ports).

OpenBSD ships spamd, a greylister which also tarpits.

PF has the max-src-conn-rate state tracking option which is commonly use to blackhole scanners:

  pass in on egress proto tcp to port ssh keep state \
    (max-src-conn-rate 10/60, overload <scanners>)
  block quick from <scanners>

Rather than blocking you could instead redirect new connections to a tar pit by replacing the block rule with something like

  pass in on egress proto tcp from <scanners> to port ssh \
    divert-to 127.0.0.1 port 2222

where your SSH tar pit service listens on 127.0.0.1:2222. This is basically the reverse of how spamd works, which diverts all inbound SMTP connections not in the <nospamd> table to spamd. When spamd's greylisting rules are satisfied it adds the sender's IP to <nospamd>.

IPv6 poses a problem for selective tar pitting, though. Realistically you need to tar pit at least /64 subnets, but even then it's not difficult to get your hands on a /48. But if you lumped /48s together you'd have a huge false positive problem. Also, AFAIU PF's state tracking capability can't track subnets, anyhow.

I must not be understanding something...

Does this still allow whitelisted machines to connect, or is this just a troll thing to do?

Presumably legitimate access would be on port 22, whereas only bots would hit 2222.

almost, but backwards. The bots all know port 22. Only you know which random other port is the real ssh port.

And the second guess would be port 2222, so don't use it for a real ssh server!

I suspect that bots that randomly scan IP addresses for vulnerable hosts don't bother trying anything besides 22. I've been running my sshd on port 622 for a long time and I never get fishy connections (while I'd get at least one a week on port 22).

On the other hand a dedicated attacker who really wants to pwn your server will just scan all the ports and figure out what is listening and where. At this point you're better off implementing some form of port knocking if that's a cause for concern.

That being said, running sshd on port 2222 is probably not a good idea because it's not a privileged port.

Every machine I control has sshd listening on port 222 for that last reason.

I use 12345 as my SSH port (not really)

The idea is that you run your real SSH server somewhere else, like on a different port. If you scroll down a little (like, skim over the article) you see some example code that is basically the entire server. It's just a few lines of code, there is no real SSH implementation in this thing. And that's the whole idea: it can't really be hacked and it thwarts attackers.

This will not allow any machine to connect on the port it's listening on.

The idea is that you set Endlessh on your server's port 22 (standard SSH port), then configure "actual" SSH to listen on a different (randomly selected) port. You connect to that port to get stuff done. Bots that troll for connections on port 22 will get stuck on port 22.

I could see a few ways to use this. For me, what makes sense for my public sftp servers is to put this on a higher port, then use ipset lists to dynamically add bots to a DNAT rule and send them to this instead of the proper sshd.

i.e. you get 2 chances to authenticate correctly, then I put you in this hamster wheel for a day. Hamster wheels and intermittent fasting are all the rage these days.

I was thinking of something similar, but in reverse -- turn this into part tarpit, part honeypot.

I was thinking I could take the IP address of anyone who hangs out in the tarpit for longer than a minute or so and automatically add it to my firewall's blacklist.

I used to run an honeypot.

For fun and giggles, I also kept the user and password they tried to see if any of my systems was at risk

Yes, it's just a troll thing.

Could the return strings be generated by a Deep Learning powered fuzzer?

What are you trying to achieve?

Stickier tar.

He's trying to crash a script-kiddy's box with fuzzing. It's a fun thought, though I'm not sure it's practical as you'd only get a few tries. Fuzzing is typically an approach of throwing it the wall and seeing what sticks; if your deep learning found something that crashed most ssh clients that would just be a zero-day.

It doesn't have to crash them all...just the one it's working on.

Maybe deep learning technobabble that might convince a script kiddie into "researching" what was coming back?

The return strings could be anything, provided they fall under the max length in the spec. DP fuzz it all you want.

Beautiful!

Ewww, sticky service! Hilarious! :-D

Consistency is a virtue.

'Tacky' is a consistency. O:-)

I'm not sure I like this. How secure is it? Anything you put on port 22 is going to be attacked relentlessly. I think I'd rather have OpenSSH handling those attacks than some tarpit program that may or may not have been subjected to the audits and the trial-by-fire that OpenSSH has.

A program like this has vastly less attack surface than a full SSH protocol implementation. In particular, it doesn't support the part of the SSH protocol that allows executing commands on a remote host. So the only way this will be less secure than OpenSSH is if there's some memory unsafety or a leak of some kind.

almost all of the SSH protocol isn't exposed until you authenticate, and in the vast majority of setups that means you get a shell to execute commands with anyways, so while still important, the key problems are with pre-auth security.

Sure, but "isn't exposed" is very different from "that functionality is physically not present in this executable."

Any tarpit has the potential to piss someone off. I'd run it on a sacrificial server with no obvious way to tie back to who is running it.

Why worry about pissing off the kind of person who scans for open SSH ports for no apparent legitimate reason? Are you concerned that the Chinese are going to try even harder to hack your machine, like they're not already doing everything they can to break in?

The vast majority are going for the low hanging fruit. I don't have to outrun the bear. I just have to outrun you.

They may be the same type of person that would launch a ddos, for example.

I'm not concerned about this, personally.

On my home server, there are only two ports open to the internet -- one for my SSH server, and one for my VPN server. Both of those are running on nonstandard ports.

In order to DDOS me, an attacker has to find those ports, which would require a comprehensive scan (increasing the chances that they'll get blocked by my ISP or detected by my antiscan monitor) or a huge amount of luck. And even then, they'll just be able to DDOS my SSH server and VPN. That wouldn't be awesome, but wouldn't exactly be disastrous.

If your IP doesn't change that often or is static, I'm curious how much Shodan knows about your open ports?

You can change your ports but it won't prevent "Eve" from scanning for those as you mention yourself. I think this belongs to the realm of "security through obscurity" tactics and won't be safe in the long run.

I can prevent being blocked by your ISP by adjusting the timings etc. of my scripts.

I think it's far more important to take a look at the supported key exchange algorithms etc.

You are correct. This falls into the cost/benefit calculation of security. A DDOS attack on my home network would be annoying, but not devastating, so I'm willing to accept a little vulnerability to them in exchange for other benefits.

> I can prevent being blocked by your ISP by adjusting the timings etc. of my scripts.

But that won't let you escape detection by my system's own defenses. Well, you could if you slowed down the scan enough -- but "enough" is pretty darned slow.

You can always tie it to the organization who is running it, by the simple fact of looking at the IP address that is "acting slow".

Years ago when I was playing network admin for a company, I set up a tarpit system which ran for a few years. In that time we never encountered one pissed off person in person (or on the Internet).

It's as safe as it can get. None of the versions he showed in the post actually read any data from the attacker. It only accepts the connection, then writes random data to it every 10 seconds forever.

I'm not sure I catch your drift. It seems like the point is that it doesn't actually implement the SSH protocol- it just exploits the fact that a real SSH server is perfectly within its rights to send any data it wants to before a version string. (And on top of that, it looks like it doesn't even read anything from the socket?)

Is there something I'm missing here (probably related to poll(2)) that could cause this to be insecure?

> How secure is it?

Since it doesn't actually have the ability to do anything, it's pretty darned secure.

The source code in the article shows that your concerns aren't really.. concerns.

Reply on news.ycombinator.com