Brute.Fail: Watch brute force attacks fail in real time | Hacker News Reader

Brute.Fail: Watch brute force attacks fail in real time

brute.fail

728 pointsmike_d259 comments

So funny story, for a while I worked on a 'reverse' exploit. Which is to say morphing the response from ssh to the client with large malformed packets. The idea was to crash the client making the request. In my case I found these attacks would have like 6 to 10 attempts from the same source address. By time stamping the requests, I could evaluate if the next attack from the same address came more quickly or more slowly. I then had my server "morph" the return payload somewhat randomly and keep the three responses that caused the most slowdown. When I got to 100 variants that had "won" this selection criteria I took the three best and started over from there. After a couple of months of this I finally got a response where after one request I would not get a second.

I felt extremely pleased with myself for about another month, and then my server address got hit with a massive DDOS attack (for me anyway) over my 6MBPS DSL line. So clearly I had hit a nerve somewhere :-). Anyway, I moved my server to a different address and used fail2ban to just note source IPs and put them into the IP tables as banned addresses. That works great and hasn't resulted in the same sort of drama as last time.

Disable password authentication and fail2ban becomes completely unnecessary.

You still get the attacks and they show in the logs

Also, it’s not just ssh, there are brute force attacks for pretty much any service that you run

Our logs show automated attempts to run exploits on our web servers everyday

This. I really do not understand why people use fail2ban when the threat is somewhere else.

It won't stop a ddos but will certainly, at some point, prevent you from logging in.

It keeps the logs cleaner. If you either don't look at the logs at all, or have fancy log analysis systems, then it does not matter. But if you are in the middle, and just manually look at the logs every once in a while, this would be a great help.

If you are logging in via ssh, the chances of being locked out arr low - using password auth is a bad idea, and once you set up ssh keys, the connection will always succeed. And in case of rare event like new system setup, you can always ssh via some other system, -J is great for that.

> the connection will always succeed.

Not from my experience. If you have too many keys and certain ssh agents like gnome keyring don't pick up the key intelligently and will try a all the keys in some order often resulting in the server giving rejecting you due to too many failures.

Yeah, this is annoying.

But you don't live with it -- you either move the extra keys to subdir, so that gnome keyring does not pick it; or use "IdentitiesOnly yes"/"IdentityFile foo" in .ssh/config to restrict certain hosts to certain keys (and yes, those work with ssh agent caching too).

I know many people just don't care about working tool, and tolerate the pain, but hopefully if someone knows enough to setup fail2ban, they should also be able to setup ssh config. Especially since reliable ssh connections is such a high quality of life improvement.

Yes I've hit this as well! Such an annoying behaviour.

However I think it's a good habit to make records in `~/.ssh/config` for each of your servers anyway just to keep tabs what, where, who, and with what keys.

> It keeps the logs cleaner.

I move my port somewhere else to reduce the clutter

> manually look at the logs every once in a while

I imagine that this is more by curiosity than anything else (which is a perfectly normal reason to do that - I do it from time to time just because). For logs analysis and alerting I have automated systems.

> using password auth is a bad idea

It is not if the password is correct, security speaking.

I disable password authentication and use fail2ban. It's unlikely they will be able to brute force my key, but no server is perfect. sshd might be compromised one day. I’d rather have an extra layer of defense just in case.

It's impossible they will brute force your key if you have a decent length.

While I'm sure it is possible for some (mainly government) actors to brute force keys, I'm also sure these do not include the same low-hanging-fruit vandals blasting brute force attacks. And I'm also pretty sure you're not one of the select targets of these highly advanced actors.

A vulnerability in sshd is indeed possible and happens once in a while. Fail2Ban won't stop this though because a known exploit will let them through on the first attempt.

I personally view fail2ban more as nuisance control when it comes to SSH with password auth disabled. Minimizing the log crap, the wasted CPU resources by the failed handshakes. It's not really a security protection in that scenario. In other cases (e.g. web logins where passwords must be used) it of course is.

Unless the maintainer of your distro's ssh package accidentally introduces an error that reduces the number of possible keys to, say - 32,767 total possible keys.[1] That's a brute-forcible number of keys that fail2ban would help mitigate.

(1: https://research.swtch.com/openssl)

Or the NSA subverts a cryptographic standard in order to produce predictable seeds for cryptographic random-number generators that are used to produce private keys. [1]

(1: https://www.bbc.com/news/technology-24048343 )

INCONCEIVABLE!

You keep using that word. I do not think it means what you think it means.

20 years ago port-knocking was supposed to solve this issue for good but it seems to have been never really been taken up. I'm not sure why.

Because it doesn't solve any real problems if you have ssh password login disabled and filter out login failures from logs

With port knocking or a simple as having SSH on a non-standard port, the connection request stops before even opening the TCP connection. That's less load on the system, less logs, less writing to storage, etc. Less of what you don't want must surely be preferable.

…all for the cost of vastly increased complexity. You’re just micromanaging your pets there. If the load of failed login attempts or the log writes brings your servers down, you have a whole lot of other problems to take care of.

Are we talking about ssh on an ESP32 or a 2U server? You'd saturate my transit port before I noticed the load on my server or the logs that I filter by default.

btw TCP is cheap compared to public key crypto

This is really only effective for password based logins, but most logins these days are key based

No, it cuts the DOS resource usage by a factor of 4 because a TCP connection is never opened.

Port knocking as it's usually done is easily sniffed. Perhaps using a dynamic TOTP-like time based seed to constantly rotate the ports might help. But it sounds overly complex.

It feels very like a "key under the third plant on the right" kinda thing. Not a solid security measure.

Single packet authorization. It's like the server is not even there unless you send a cryptographically signed packet

https://cipherdyne.org/fwknop/docs/SPA.html

Nice solution. I hadn't heard of it. I immediately thought of replay attacks because it's a one-way protocol but it looks like they mitigated those as well.

Moxie Marlinspike made something to address the simple knock server's problems.

https://archive.is/MZKkb

This is fine, though. "Security through obscurity is not security" but moving your SSH port to something not 22 will utterly eliminate brute force attacks.

It's too much bother to go find it, and the bozos will just move on to the next machine with port 22 open.

That's still not really security but just a nuisance mitigation IMO :)

It's both, really. If you're not getting scanned, you've reduced an attack surface, and that can only be good.

Well, I view it as hiding an attack surface. It's still there, just harder to find.

But I know I'm a bit of an absolutist on security.

You put a lock on your bike.

But you also put it in the shed, and lock the shed.

Nuisance mitigations are part of security too! Fewer irrelevant notifications makes it more likely you’ll notice when something really is a problem.

It’s like how an adversary might launch a DDoS attack at the same time as they exploit a SQL injection vulnerability to exfiltrate credit card information. Filling up logs and alerts overwhelms the blue team and makes it harder to notice the quieter, but more dangerous attack.

Security through obscurity actually is security, and is perfectly valid to use with a defense in depth strategy. The problem is when obscurity is the only defense.

I agree, these attacks are looking for systems that have pretty default security, and by running on a different port you avoid all this automated chaos because you're non default now. Like scam emails with typos, a way to filter out the naïve people.

> This. I really do not understand why people use fail2ban when the threat is somewhere else.

Fail2ban keeps my log short enough that I can review them daily, I don't have to sift through thousands of login attempt.

> It won't stop a ddos but will certainly, at some point, prevent you from logging in.

Yup, losing my key and having no password access will do that.

I'm not sure manual log scanning is all that interesting. You probably want to produce a list of successful logins and review that regularly.

If an attacker manages a succesful login you've already lost. Better to catch them in the act while they're scoping you out.

If your SSH key leaks you're not going to have a warning. All you'll see is a login from yourself that you don't remember.

I am pretty sure we turned off password authentication like 10 posts up this thread.

This is why I require both a private key and a password.

I have fail2ban configured to block IPs with invalid private keys after a couple attempts, and if the key is valid to email me and rate limit invalid password attempts.

This gives a more than sufficient warning if my key leaks which is already very unlikely, and this just makes it much more unlikely for both to be compromised, and only took an extra 5 minutes to configure.

How do you configure emails on successful logins? Can you share your config, sufficiently anonymized?

I created a jail for fail2ban with

  logpath = /var/log/auth.log

and for the filter I use

  failregex = .*Connection closed by authenticating user [a-z_]([a-z0-9_-]{0,31}|[a-z0-9_-]{0,30}\$) <HOST> port [0-9]* \[preauth\]

and the emails are just cron with a python script that checks /var/log/fail2ban.log for any new Found|Ban|Unban IPs and sends them using smtplib

If you like I can share the full config files but the rest isn't too interesting nor different from what is here https://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Jails

I move my port to an uninteresting place and it considerably reduces the amount of logs. If this was a nuisance I would filter out the messages

> Yup, losing my key and having no password access will do that.

You can also lose your password. Or forget it if you know it by heart. But in any case you can use a password instead of a key - it just needs to be good enough (= long and not in cracking dictionaries)

I really don't understand why people use keys when the threat is somewhere else.

Assuming that your comment is serious, where is the threat in your opinion?

Bad passwords.

Then I do not understand. Keys are the answer to bad passwords (among other things)

It is one answer. Good passwords is another.

I prefer libshield but one thing I've found is that annoyingly sshd didn't use PAM to check if the login user is valid apparently (so it never fires when only using keys).

Not really. A lot of these bots are super duper dumb and continue slamming your server with handshake attempts even if password auth isn't even turned on. Every handshake is wasted CPU resources (and asymmetric crypto isn't cheap). It also makes it harder to see real dangers in the logs.

And/or run sshd exposed only to a wireguard or tailscale interface.

Not for other services... I have fail2ban parsing my NVR logs. Wrong password three times and it permanently blocks the IP on my Opnsense firewall.

And change the sshd port. I use my birth year. 0 brute force attempts in the logs after that.

You still got the response packets?

Nice. Also an illustration that its good to establish one's inner motive first, like here might range from a desire for simple solutions, to another dimension of drama, which might indeed be desirable sometimes, for entertainment or for snapshots to use in a book one is writing. :)

Another fun one is sending a redirect to a gzip bomb or extremely large file (i.e. speedtest download file) for when certain non-existant URLs are requested, i.e. /wp-login.php on a non-wordpress site.

Earlier this month I was downloading top 1 million websites' robots.txt and some guy just served 100mb of whitespace :/

I love it!

Once I was trying to scrape from some internal API, and made a request that used parameters the real client wouldn't make. I didn't expect curl to scream "FUCK YOU <some name>" at me. Presumably the name belonged to someone who scraped it before.

LOL, that is great.

Is it legal to host a gzip bomb? Seems like a malicious file to serve up…

It can cause your site to end up on Google's safe Browsing black list which can be a death sentence for a business. Google has automated process for identifying malware and black list such websites. Almost all browsers use this list to warn users. This is why it is also dangerous to host anonymous uploaded files even for a short time. https://news.ycombinator.com/item?id=25802366

> It can cause your site to end up on Google's safe Browsing black list

If it's a file that nothing on your site links to, and doesn't really "exist" how would google ever index it? Especially if you put in as a deny in robots.txt, which as far as I'm aware, Google honors.

Distributed valgrind

Try negotiating ssh compression and then sending terabytes of compressed zeroes. Might work for a badly implemented scanner.

Semi-related, I've just started serving large files to webserver scanners, some are so poorly made that they just download the entire file

Please tell me it's a massively upscaled hello.jpg you send...

Details!

Seriously this is both hilarious and intriguing and deserves a long form blog post or something.

Seriously, this should be standard practice. Fail2evil...

Wow that's a cool evolutionary algorithm in practice! I'm actually surprised it worked, especially with the amount of fuzzing I'd expect ssh ecosystem to receive (well perhaps the servers do, but not the clients haha).

MySQL has a weird feature, by default enabled in many clients, that the server can request a file from the client.

I have requested /etc/shadow, cracked the hash for root password, and ssh'd back into the botnet node that was bruteforcing passwords. I then shared the information with the webhoster where the botnet was running and a local infamous antivirus company (Avast before it was leaked that they are evil) and got a t-shirt.

https://www.abclinuxu.cz/blog/jenda/2019/2/exploiting-mysql-...

Outstanding job! My opinion of mysql continues to improve :-/

As a curiosity, I found the submission link in the comments on that thread https://news.ycombinator.com/item?id=19305823

Kudos to 185.65.135.x, who attempted to log in with username hn_i_found_it right as I checked HN after work and opened the link, and who proceeded to attempt a number of different simple SQL, html, and js injections. I approve of this effort. I also approve of the author of this site, which appears to have survived this minor attack! :D

That’s neat. What’s the total volume per day? Are the passwords themselves being escaped in the final UI rendering? Otherwise you’d have an XSS for a password like “<script>/* code */<script>".

EDIT: Unless it's happening on the server side where it's being saved, I don't think they're being escaped:

    col1.innerHTML = '<span class="fi fi-' + msg.cc + '" title="' + msg.cc + '"></span> ' + msg.src;
    col2.innerHTML = msg.proto;
    col3.innerHTML = '<code>' + msg.u + '</code>';
    col4.innerHTML = '<code>' + msg.p + '</code>';

It is escaped server side. Anything long enough to be a useful payload is trimmed.

How short we talking?

Since there's multiple opportunities to inject code, it's possible to split out the payload across multiple fields: https://www.highseverity.com/2011/06/xss-in-confined-spaces....

Ten characters per block is enough for:

    <script>/*
    */eval(/*
    */'....'+/*
    */'....'+/*
    */'....'+/*
    ...
    */)/*
    */</script>

Best to escape everything at render time.

Everything is escaped server side before it is sent to the client. Shoot me an email and I will let you play with it after the HN traffic dies down.

As the other comment said, just write proper code instead of going "oh but it's escaped and size-limited."

JS has sane APIs where no string is "dangerous," use them.

Sorry, but this is not the way. It's like saying, "but I am escaping my inputs on sql with my function"... instead of doing the right thing.

The equivalent code is really not that hard:

    const span = document.createElement('span')
    span.setAttribute('class', 'fi fi'+msg.cc)
    span.setAttribute('title', msg.cc)

    col1.appendChild(span)
    col1.appendChild(document.createTextNode(msg.src))

    col2.textContent = msg.proto

    let code = document.createElement('code')

    code.textContent = msg.u
    col3.appendChild(code)

    code = code.cloneNode(true)
    code.textContent = msg.p
    col4.appendChild(code)

or something in these lines.. you get the idea

For this reason I've put `endlessh` on port 22 and moved actual ssh elsewhere...

Also started using Crowdsec recently, but not sure about if it's worth it...

fail2ban out of the box works fine for SSH, but for dovecot and postfix it's somehow broken, and the configuration scripts are just too obtuse.

Nice idea. From the docs:

Endlessh is an SSH tarpit that very slowly sends an endless, random SSH banner. It keeps SSH clients locked up for hours or even days at a time. The purpose is to put your real SSH server on another port and then let the script kiddies get stuck in this tarpit instead of bothering a real server.

Since the tarpit is in the banner before any cryptographic exchange occurs, this program doesn't depend on any cryptographic libraries. It's a simple, single-threaded, standalone C program. It uses poll() to trap multiple clients at a time.

https://github.com/skeeto/endlessh

Wow. That is pretty brilliant.

That’s hilarious

I'd be cautious about stuff like this - if you annoy the wrong person that could paint a target on your back.

Ah where's your sense of fun ;)

Realistically though, they'll probably timeout by themselves automatically if they haven't seen a password prompt after N seconds. TCP connections can hang overall, so having that would be basics anyway.

Don't discourage someone else from doing it and risk their not coming back to update us on the results. This popcorn is not going to eat itself.

They're impotent little weasels who couldn't hit a target the size of a barn, so the risk is minimal.

"do not guess that it will be your weakest opponent nor guess what they will try, prepare for your strongest adversary and all that they can do" -- mutatis mutandis, Sun Tzu, Art of War

Not to disagree with infallible thousands of years old advice, but surely resources are in far shorter supply than potential attacks, so we have to prioritize and filter.

If this was default on port 22 of every cheap router sold, and if you enable ssh it has to go on some other port, it'd frustrate a lot of bad guys.

I spent 10 minutes to set this up. I was shocked to see that I got my first taker less than a second after I opened port 22 on my firewall.

https://www.abuseipdb.com/check/178.62.237.183

Unfortunately, it only wasted 30 seconds of that IP's time.

It's not clear what type of tarpit would waste the most of the operator's time. Maybe something like a "byzantine VM", that seems exploitable, takes payloads, passes initial checks, and then starts having "problems". DDOS attacks redirect to the C&C server. Coin miners report false mined coins. Hosted files have corruption, and won't complete transfer, etc. Whatever it is, it needs to somehow seem like the operator has an error in their code :)

Remember the guy who asked ChatGPT to pretend to be a terminal, you could probably do something similar to mess with SSH intruders.

Reminds me of a tactic I tried with telemarketing scams when on paternity leave. I got so many calls like "Your Social Security Number is about to be revoked" that I developed an entire persona with a name, address, SSN, income, debts, and other factors. I spent several baby-naps looking up jobs, apartments, car loan interest rates, and other factors to make the persona as realistic as possible. My wife talked me out of buying a $5 Visa gift card with cash so I could give the scammers a credit card number and keep them talking longer.

That seems like overkill. I just disable password authentication, and use SSH public keys only. It prevents brute force attacks completely.

This, and move the endpoint on an uninteresting port to lower the noise in the logs

... and avoid automated attacks in case a 0-day in the pre-authentication OpenSSH server code shows up

If there’s a preauth RCE in ssh, I’m sure threat actors will start scanning non default ports once their hornets have hit critical mass.

I suppose it may do that, but naïvely switching to an alternate port may lull one into a false sense of security: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/SSHAltPort...

Shodan will find and fingerprint you easily enough.

In theory. I have a server running on a weird port, but with fail2ban just in case. I haven't had even a blip of an attempt on it.

It's not for security, it's public service and entertainment

Do the dovecot and postfix scenarios work well for you with crowdsec? Always like hearing user stories :)

I think the blocklists have filtered out a bunch of that noise, so from a straigh-up dovecot logs I've not seen much yet...

I setup SSHD to listen on a wireguard interface rather than listening on all interfaces. This makes SSHD only accessible to wireguard peers rather than the entire internet.

A nice aspect of wireguard is that it's "steath", meaning that it does not respond to unauthenticated connections at all, so there is no way to probe and scan for wireguard listeners at all.

I think setting up daemons behind wireguard offers a lot of security. SSHD is probably fine to expose, but something like an IRC bouncer for example really benefits from being protected I think.

I use ZNC over wireguard for this reason! It also allows to use ZNC securely without the need to setup TLS certs, which IME is actually harder than setting up wireguard!

maybe telnet over wireguard ? :)

That just sounds like SSH with extra steps

... except that SSH isn't "stealth".

Ah, I see what you mean now

> A nice aspect of wireguard is that it's "steath", meaning that it does not respond to unauthenticated connections at all, so there is no way to probe and scan for wireguard listeners at all.

I did not know this. That’s really cool.

Is it done over a stateless protocol like UDP, or is a TCP connection opened first? Ie. is it impossible to see if there’s even a server there at all, or is it first revealed that there’s a server accepting a TCP connection?

It's over UDP, you can't probe for wireguard AFAIK

depends on your firewall, lots of setups give you an ICMP port unreachable in response to probing closed udp ports so you can often tell a wireguard host if the default port doesn't (in practice)

openvpn has had an option for such behavior (over udp obviously) for a while. the option is called tls-auth, it requires you to go through and generate an aditional key which has to exist on all clients and server. Last i remember is that even if you scan the server, it is completely quiet unless the right signature is received as well for each frame.

This is why I come to HN - brilliant ideas that I should have thought of before, but didn't!

I just changed my server, and uninstalled the - now truly useless - fail2ban. I use SSH keys of course, but without fail2ban my server's logs were constantly flooded with hacking attempts.

No longer - wireguard for the win. Thank you, chlorion!

Its just stupid at this point. You can put up a wordpress site with 5 daily visitors, and you'll still get thousands of SSH / xmlrpc hits per day.

The range of IPV4 address's is small enough that a single server can scan it, in a single day.

So I guess the problem isn't going away.

Man, that one Brazilian IP really going hard

it bugs me that they're not trying the passwords in lexigraphical order :-D

also, who has sshd without `PermitRootPassword=no`? they need to broaden their horizons and try `admin`, `ec2-user`, and `ubuntu` /s

Who still allows password-based login for any SSH account, root or not? Keys, certificates, or Kerberos for all users.

You'd think, but yet one still sees one or two posts a week to SO or whatever forum asking how to enable password-based login to AWS EC2 instances. Often with screenshots showing '0.0.0.0/0' as the allowed incoming range.

Sometimes the default after creating the VM. Sometimes it is, if you start in the recovery mode (usually with a random long password)

I do. For some reason ssh keys became the group-think security advice to repeat ad nauseam. I often find people have only considered this very shallowly, and their reasoning is just "But OMG, entropy lolz" without actually seriously considering the available entropy and likely attack vectors and failure.

Why?

The benefits are largely theoretical if you choose sufficiently strong randomly generated passphrases, with some symbols and numbers mixed in, which I set my password manager to do. (Actually, I have a couple of super important, super long ones I keep only in my head.)

The draw-backs however are several:

1. I find that its not uncommon I find myself having to chain together ssh tunnels and other strange things to debug networking issues and need to login to another machine from a new machine that doesn't have the ssh-key. Treating servers as cattle exacerbates this issue since it's more likely to occur.

2. If a machine or server I use to manage stuff as a gateway/proxy/vpn entry to my network has all my ssh keys on it, and it becomes compromised because of some 0-day, the attacker now basically has access to... multiple entire networks. Now, this is technically true of my password manager as well, except that the context / IP addresses, etc, is not going to be as obvious as it will be on the server where the attacker can see the running services, check logs, command histories, etc. With the password-based management, sure I could get hit by a keylogger, but everything won't be compromised by default.

3. In my experience, you're more likely to lock yourself out than let an attacker in. You basically still have to have your keys managed via some type of password manager anyway because otherwise you run the risk of getting locked out of everything in case you're away from your primary machine and need to address some emergency or something.

> ... need to login to another machine from a new machine that doesn't have the ssh-key.

> gateway/proxy/vpn entry to my network has all my ssh keys on it, and it becomes compromised because of some 0-day, the attacker now basically has access to... multiple entire networks

That's why you use SSH agent forwarding https://docs.github.com/en/authentication/connecting-to-gith..., so you never need to copy the private keys to other computers, and thus they can't be stolen from there.

On the other hand, SSH agent forwarding means exposing your SSH agent to the bastion host. If that gets compromised, an attacker may be able to move laterally to other systems your personal computer had SSH keys for.

Just use `ssh-add -c` and confirm each use. If you have _really_ old ssh (which is probably insecure anyway) using gpg-agent as ssh-agent had this feature a few year earlier, but we are talking archeology here.

You can also use ProxyJump to let SSH handle setting up tunnels for you. It logs into each hop via the tunnel(s), no need to forward your keys. Great thing is, once your ssh config is properly setup you don't care how convoluted the tunnel setup is; ProxyJump will connect to jump hosts via other jump hosts.

...May not be stolen, but may be abused from there via ssh-agent connection.

All of your items are just "i'm doing this wrong" combined with not understanding how keys even work.

Your private keys shouldn't even be accessible to you, they should be on a secure enclave like a yubikey, and you should forward the token along the chains. No risks, and basically painless, especially if you switch to certs so you don't even have to know the public keys ahead of time on the servers, just all trust the same private PKI.

> Your private keys shouldn't even be accessible to you, they should be on a secure enclave like a yubikey

That’s not the only reason or way to use SSH keys. Even when stored on disk (in plaintext or encrypted) they offer advantages over password authentication, e.g. making it impossible for an MITM to steal credentials or impersonate an authenticated client even without validating host keys.

GP is deeply concerned about getting locked out, and your solution is to use a secure enclave?

with certs its not a problem to swap keys out.

> But OMG, entropy lolz

Public key authentication isn’t just providing more entropy than passwords.

Passwords (as used in SSH) are bearer tokens – send yours to the wrong server, once, and you‘re compromised, for this and future sessions. That’s not the case with public key authentication.

If you think a key is just a longer password, you don’t understand public key cryptography.

A man in the middle attack where you accept the server key can steal your password, but it can’t your key.

I do on my local network. Nothing that's on the Internet. I think it's still enabled by default on Fedora and Debian.

The people with PermitRootPassword=no are also the ones that’ll have weak default password. It’s probably actually saving the attackers time that allowing root password login and bad password choice occur together. If everyone with randomized high-entropy root passwords actually allowed root password logins that bruteforcers would have to spend so much more time!

Even if you disallow root login with a password, the user can still try to log in, and the attempt still gets logged

If you're asking why it would ever be worth it, there's always valuable stuff online with incompetent configuration. I don't know if shodan is still up, but I remember going on there in high school and getting access to random webcams (sometimes in peoples' homes)

https://www.shodan.io/ is still up, if that's what you mean.

I do on my home servers where I am the only one and do not need traceability?

Why do you ask, because it is dangerous?

I would love to learn something here.

Yes, someone got a word list. Shame it's a Chinese one.

That's what a typical attempt looks like unless they're progressively timed out with fw blocks

This is a pretty excessive amount of SSH brute force but it honestly doesn't seem as bad as some of the cloud machines I run. There are always like at least 3 different IPs going relentlessly hard 24/7/365.

I have an RDP server open to the internet(on a custom port) and it just receives an absolutely relentless stream of login attempts with all kinds of random logins. That's a private server on a private home IP, not associated with a known domain or anything. Changing the port stops it for about 24 hours then it starts again.

Yeah, we have the same problem. I made a custom firewall rule and a python script that watches the windows logs for multiple failed logins to combat this and it seems to work pretty well but there's always new ips.

IPBan does that:

https://github.com/DigitalRuby/IPBan

Oh I didn't know this existed - will definitely set it up, thank you!

https://github.com/al-ign/Protect-RDSFromBruteforce

I know nothing about networking, so pardon the ignorance:

Why isn't there (or is there) some kind of service you can use to map some crazy URL to your personalip:port, like...

http://obscuremyshit.com/393nnasjhf83u98723401 = personalip:port

And only when a connection is referred from that source, does the RDP server even expose itself? And for all other traffic that hits personalip:port, it does absolutely nothing?

There are numerous ways ways around it (port knocking, VPN) that gambiting (for whatever reason) is choosing not to employ. Your idea is a good one though

Is there some referrer field in the RDP protocol handshake that I'm not aware of?

The issue in GP's idea is, that there isn't such a referrer field (his first sentence is correct). And even if RDP has it, the service would be limited to RDP, and GP's idea is probably to have it for any service thinkable, e.g. also SSH.

A solution on top of that would be for this obscuremyshit.com service to have a client-side listener. So if I get a hit on the page obscuremyshit.com/393nnasjhf83u98723401 from IP x.y.z, the obscuremyshit client running on the target cimputer gets a signal from the obscuremyshit server to "Open port 22 and allow a connection from the IP x.y.z" (I guess that means the client would have to be in control of the firewall rules), and then the computer with the IP x.y.z would have to establish a connection within a timeframe.

Of course it would get more complicated if e.g. the above URL is hit from a different IP address (e.g. from someone's phone over 5G, and the SSH connection wants to come from a laptop over a cafe WiFi).

>Why isn't there (or is there) some kind of service you can use to map some crazy URL to your personalip:port,

You might be able to do this by responding with a hyperlink which points to ssh://x.x.x.x:1234

Some browsers will recognize that format and pass to an ssh client application to spawn an appropriate ssh connection.

The reason why it's not a worthwhile idea is that there is a limit of 65535 ports to chose from on a given IP address and they all can be scanned pretty quickly in order to locate an active SSH service port. This really makes the URL idea ineffective.

Port-knocking may be a little bit more effective because the SSH service will not reply to a connection request unless you've attempted to establish a connection on a different port first. Scanning for an open SSH service becomes much more difficult.

That’s how I hide my password manager, but that’s HTTP and has Header fields to dispatch on in the reverse proxy.

Why not VPN?

Because it's a disposable server running in an isolated VM that I need for one reason only and even if someone does break in(impossible with these random logins, and I assume RDP doesn't have any currently known security faults) then it wouldn't be the end of the world - I have a notification on successful login so I'd be told if it ever happened and I would just kill the VM instantly. Right now it's exposed to the internet for simplicity sake.

Welcome to the world of "cracked RDP"! Various internet lowlives get a big list of bruteforced RDP IPs/credentials then sell them in bulk: https://www.bleepingcomputer.com/news/security/over-85-000-h.... Sometimes the hacked machines are used to start attacks from.

Oh that’s pretty fucking cool.

Is code available anywhere?

I run a slowly growing network of SSH honeypots that do central logging (Greylog), that I’ve been meaning to document the setup of for here somewhen.

Bolting something like this onto that would be pretty funny.

No code yet, it was literally pieced together last night.

It is only processing SSH attempts from 3 hosts right now (one in colo, one EC2, one DigitalOcean) because when I pointed the full firehose at it the user experience of the website wasn't great.

I put on my firewall a block for incoming traffic for all IPs outside Europe which helped a lot with the quantity of attempts.

Very naive question: can a bad actor just use a VPN to get around this?

Yes

Not if you also block VPNs

Fingerprinting VPNs is quite hard so this isn’t a reliable solution.

There's plenty of services available that provide reliable VPN and proxy detection. Every large corporation uses them.

is it possible? how did you filter?

(I've tried 2 years ago: at that time even aws is not able to provide a reliable EU ip list)

I use this setup [1] on my servers. IPs are mapped to countries using Maxmind's GeoLite2 database. Linux's Tcp Wrappers are configured to block access for all IPs that aren't in my country.

A custom fail2ban jail adds all IPs that get blocked by the Tcp Wrappers to the system's firewall.

[1]: https://www.axllent.org/docs/ssh-geoip/

thanks!

People still use TCP wrappers? I thought that went out of style 20 years ago.

I like that I'm not alone in the world of obsessively looking at my log files.

This site is now brought down due to the collective brute forcing of HNers visiting the site, as if akin to a DDOS. How ironic.

Is it not working for you? I see over 350 active users at the moment.

I did get one report that it was blocked by a corporate network because the domain was newly registered.

The Websocket connection did stop updating a few times when your site was getting the HN hug.

You could push an updated list to Cloudflare KV every few seconds and have the front end poll it from there for cheap and near unlimited scalability.

I've seen people call this the "Hug of death" when your post gets so popular the site goes down

Damm, the same brazilian IP is trying really, really hard

If this doesn't get to install fail2ban don't know what will.

And SSH key only login

I was actually thinking about that: OT1H, fail2ban would really clean up the list, so it's not monopolized by the one joker, but OTOH given sufficient spans of time it would make the output go quiet, which for this specific case defeats the purpose

I actually much prefer the projects that give the caller a fake shell, and watch what they type after "breaking in." It'd be the Kitboga of ssh attacks :-D

I would 100% watch the Kitboga of ssh attacks, is that something that exists today? The closest I've seen so far is password purgatory - https://www.troyhunt.com/sending-spammers-to-password-purgat...

> give the caller a fake shell, and watch what they type after "breaking in."

Oh YES! Do it, please! We could learn a lot!

I believe one of ISC dshield's related projects can do this.

Thanks for the reference; after some link chasing I was able to end up on the project I believe you're thinking of: https://github.com/cowrie/cowrie#features (appears to be BSD-3-Clause: https://github.com/cowrie/cowrie/blob/master/LICENSE.rst )

I don't see the point of fail2ban on a server without password login, except to keep the log file tidy. That isn't worth risk of locking out legitimate users due to misconfiguration or user error. CMV.

Keeping the log file tidy isn't just an OCD thing. If you're searching for a needle in a haystack, where needle is "suspicious login", and the haystack is "all of the login attempts from the past the months", your job is made much easier when the haystack is much smaller.

That said, the fail2ban defaults are way too low and I've locked myself out with them. They can be turned way up (ban after way many more attempts) so that there's no risk of locking out legitimate users. (Assuming your users didn't forget their exact password and then generated a small dictionary to try with.) On a server with potential misconfiguration, accepting passwords is one of them.

I use port knocking on 3 preselected ports to open up ssh which is not on port 22. I can't bother to have my logs cluttered with these garbage.

I don't use password authentication and enforce fail2ban on all of my publicly accessible ssh servers.

Even if someone was to steal my keys or knock at my ssh servers with a zero-day, I will be alerted[0] of any successful login(s).

[0] https://github.com/64mb/SLAT-ssh-login-alert-telegram

A remark on your fail.js, since you’re engaging here and I figure this could interest you or others:

Once there are more than thirty rows, you fade rows in like this:

  row.style.opacity = 0;
  let intervalId = setInterval(function() {
      opacity = Number(window.getComputedStyle(row).getPropertyValue("opacity"));
      if (opacity < 1) {
          opacity = opacity + 0.1;
          row.style.opacity = opacity;
      } else {
          clearInterval(intervalId);
      }
  }, 100);

This would be better done with a CSS animation or transition—it takes less code, and is smoother.

My suggestion: use animation. Replace the JavaScript with this:

  row.classList.add("fade-in");

And add this CSS:

  .fade-in {
      animation: 1s fade-in;
  }

  @keyframes fade-in {
      from { opacity: 0; }
  }

This does behave a little differently, as if the window isn’t visible, it’ll (roughly) wait until you focus the window before playing the animation. Frankly this is even mildly more desirable.

The alternative: use transitions, which are a tad more involved because you have to trigger the value change one frame later, so that it recognises that something has changed and interpolates to it, rather than it just being the initial value and applied instantly. This JS would do:

  row.style.transition = "1s opacity";
  row.style.opacity = 0;
  requestAnimationFrame(() => {
      row.style.opacity = 1;
  });

Or you could express it with more CSS, like with this CSS + JS:

  tr {
      transition: 1s opacity;
  }

  .invisible {
      opacity: 0;
  }

  row.classList.add("invisible");
  requestAnimationFrame(() => {
      row.classList.remove("invisible");
  });

—⁂—

You might also like `vertical-align: middle` on your spin.svg.

—⁂—

On the flag icons, here’s a cool alternative technique: https://en.wikipedia.org/wiki/Regional_indicator_symbol. Lets you avoid needing even images. Unfortunately, I think Windows still doesn’t ship flags, so you’ll get the two-letter country code there. You can get around this by packaging a web font. It’d be nice if someone would neatly package a flags-only font so others can easily use it. Here’s what I did a few months back for https://ganintegrity.com/country-profiles/, resulting in a single 77KB font file:

  /**
  Copyright 2020 Twitter, Inc and other contributors
  Graphics licensed under CC-BY 4.0: https://creativecommons.org/licenses/by/4.0/
  Twemoji Mozilla packaging via https://github.com/mozilla/twemoji-colr, subset to only include country flags.
  */
  @font-face {
      font-family: flag;
      /* (Generated with `pyftsubset /opt/firefox-nightly/fonts/TwemojiMozilla.ttf --unicodes="U+1F1E6-1F1FF" --output-file=static/TwemojiMozillaFlags.woff2 --flavor=woff2`.) */
      src: url(/static/TwemojiMozillaFlags.woff2) format("woff2");
  }
  
  /* Why do we do this? Because at the time of writing Windows doesn’t do flags, so ‹guzzled by HN› will look like “ᴀᴜ” rather than an Australian flag. (macOS and major browsers on Linux do.) */
  .flag {
      font-family: flag;
  }

Thanks! This is incredibly helpful! I searched around a bit at like 3 AM while I was building it, but couldn't find a simple clean example like what you provided. Once traffic dies down i'll swap it out.

wow I just got a lesson in css for free. Thanks for taking the time!

Talking about animations, the little animation next to "Connected to WebSocket" is an SVG. I did not know until today that SVGs may be animated. Nice to know.

That’s SMIL. You can also use CSS animations on SVG these days.

I guess you guys could find this also interesting: https://infosec.exchange/@rdpsnitch

I kinda want to know the server's address so I can send a "hello_hn" message in the passwords there.

I tried the IP resolved by the domain "brute.fail" but it doesn't accept SSH connections :)

The Internet is not that big, though. You could potentially try all 2^32 IPv4 addresses with your password of choice and see when it came up here.

There's an infamous classic image out there titled "How to catch Script kiddies" which involves four nested For-loops, then placing the generated 1.1GB .txt file on a torrent site under the name "Every IP Address Ever (Hacker Tool)"

I tried mikedamm.com (the author's domain) and it does accept SSH but my failed login attempts don't show up.

Fun idea, but the data for this site (which is streamed via a WebSocket) doesn't contain that information :)

Upthread the author says one of the three servers is in Digital Ocean, which gives you an AS to target. Reverse engineer the web page so you can capture the websocket to your terminal, grep it for your IP, and search all of DO's ipv4 address space. use Shodan to limit your targets to only machines with an open port 22 to make it go faster.

Looks like it's been figured out. People are basically using it at a chat service now

This sounds interesting. When it comes to my setup, I don't even answer their connection attempts. I just log the IP and call it a day.

You'd think that would be enough for them to stop, but I have some IPs with 25k connection attempts over a 90 day span. (Of course it had to be someone using digitalocean)

It's not that funny.

While there may be millions of useless attempts, it only takes 1 to get through.

Of course SSH has a great option in key / certificate auth. So if that's enforced it's not such a big deal. Many other systems don't (at least not until we finally implement Passkeys everywhere).

It's not a real ssh daemon; there's no "through" to get.

Seems like some of them are residential IP addresses! I guess parts of a botnet or a compromised device?

Internet of things.

Lets put 20 unpatched linux computers on every network and see how that goes.

"residential proxy" is a very popular black market service, since it's less straightforward for website owners to block than other common vpn termination points. These applications market themselves as free VPN services, get loaded by special offers bolted onto legitimate software installers, or are added to trojanized pirate distributions of popular applications.

It is probably time for OpenSSH to deprecate and remove support for password based authentication.

This is fun to watch, seeing all the passwords is pretty interesting.

Just curious, why x out the IP at all?

Presumably many brute force attempts come from compromised residential PCs, whose owners may not be aware that they are participating in an attack. It's not especially polite to expose all of that personal info.

> Just curious, why x out the IP at all?

Trying to avoid being a jerk. The sources are likely hacked boxes where the owner has no idea.

And since they don't get called out, they won't get an idea. Unless the infection is retargeted against themselves.

I also scan the internet quite a bit. Trust me, they (or the ISP rather) are getting a few emails an hour.

> few emails an hour

Been there, done that. Most isps and providers don't respond or act on abuse complaints anymore.

> Trying to avoid being a jerk.

There are lists updated in real time by white hats, sharing exact IP of machines known to be engaging in bad behavior. You can, if you want, update your servers in real-time from these lists and then act accordingly: drop the traffic, reject the trafic, serve a "your IP address is participating in brute forcing attempts" page, etc.

FWIW some ISPs may be monitoring these looking for IPs on their subnets and acting accordingly.

It's not about "being a jerk". It's about giving attackers the middle finger.

It's crazy how much bots can keep trying to enter, and this bot from China really keeps trying, fail2ban really saves our ass

This is amazing!

Are you open sourcing this?

Can others stream their logs to your servers and have a crowdsourced list of attackers in real time together with their activity?

See abuseipdb.com if you care about this.

Thank you!

Wish the ISPs would subscribe to this and blocked traffic from the abusive IPs in their networks

Clicked on a random IP on the front page, it said it’s from Palo Alto Networks in Santa Clara, that it was first reported in 2022 and the last report was 5min ago. So that IP has been doing shady stuff for months and it seems their ISP (Palo Alto Networks) doesn’t really care

Palo Alto Networks is doing the scanning themselves. I see them in my logs all the time. Won’t do much good to report an abuser to an abuser. They are a “cybersecurity company” like most malicious companies.

Hey we actually built the second part as a product. Its a modern revamp of fail2ban combined with crowdsourcing aspect to deliver an up-to-date blocklist of active threats. You can check it out at https://github.com/crowdsecurity/crowdsec

What about an actual user fat-fingering their username but entering their password correctly? Would this publish that attempt publicly?

From the site:

> No legitimate services are offered on the addresses receiving these attempts, so there is no chance of a real user accidently submitting their credentials.

looks like it's getting hn-hugged to death.

Guess it got brute forced

Two hours layer, it's still dead.

Yep, totally pwned

Hugged on port 80/443 or 22

This is cool... is it open source? :)

Would be nice to have timestamps on these to get a feel for how fast they're coming in.

Some of these credentials could be of legitime users who have just mistyped the IP address.

The odds of that are slim to none.

I'm of the opinion that if they accidentally leak their credentials to anyone, they might as well leak them to everyone.

Sure, the risk of compromise goes up, but the whole culture of "we only sent those passwords once on unencrypted ftp across the internet" leads to powerful nation state spies having a field day...

If all unencrypted data sent over the internet were in a big public archive for everyone to see, then people would soon clean up their security habits.

In the intro text it specifically says:

No legitimate services are offered on the addresses receiving these attempts, so there is no chance of a real user accidently submitting their credentials. (Yes things other than SSH occasionally show up)

Would be cool to have another column true/false if that IP is a VPN IP

Is hiding the last octet of the IP address really enough privacy?

Do these people deserve privacy?

Yes, because many are residential IPs, so likely hacked machines in a botnet.

I was struck by the preponderance of Singapore. Why there?

Getting PR_END_OF_FILE_ERROR on Firefox, anyone else?

I was too.. I then disabled DNS over https (which was pointing at nextdns) and it started working. Not sure if that was coincidence or an actual fix

"Yes things other than SSH occasionally show up"

Like what?

I just saw an FTP one

HTTP/HTTPS basic auth, FTP/FTPS, LDAP, SSH, and a few other protocols are supported. I only show ones that actively try credentials, not just port scans.

Interesting!

Thinking about it, fail2ban is almost entirely a placebo given that your password should be basically impossible to brute force anyways if you have the knowledge to implement fail2ban.

It can conserve server resources to just stop responding to brute force attacks

If your server is a Gameboy, maybe.

A Gameboy would probably have the computing resources to do a thousand such calculations a millisecond.

Also disk space - i don't want to keep 500 MB of failed login attempts just to have a week of syslog available.

Rotate your logs bud.

Also, suppressing these logs is the same as rapidly rotating new logs.

It is not. Deleting my spam folder is not the same as deleting yesterdays email.

Rotating only splits the data up into N files, not make it consume less space for a week of logs.

logrotate compresses logs.

Better: just ban password logins, and use cryptographic keys instead.

Use Role Based Access Control

Fail2ban can work on more than just sshd.

Realtime honeypot

http://www.m-d.net/projects.php

Would be cool to see brute.success

Are these really failures though? This is just how brute force works.

> HELLLOOO HACKER NEWS!

Server down

be careful, this website is easy to vandalize

It's easy to see that *certain website* has not yet found this host, judging by the lack of *certain words*.

Reply on news.ycombinator.com