Block web scanners with ipset and iptables
nbailey.ca
nbailey.ca
I also have a fake /admin path that just contains a bunch of offensive/illegal phrases in 10 ish languages, but it was out of character for the post.
444 is a good idea though, I didn't know about that response code!
For others who aren't interested in reading the whole thing:
The author of the post used zip-bombs, which are compressed HTTP responses that expand to 1000 times the size of the compressed data. He could send relatively small responses that would fill the requester's memory and crash the process. Beautiful.
This distinction is important if you have a load balancer in front of nginx. The LB will wait until timeout for a response, occupying a bit of stateful memory and probably causing an error which is indistinguishable from "backend application server is offline".
On a well configured site the LB timeouts should be short enough anyway.
But it is a risk, especially on classic DOS attacks.
The other problem, if you are behind an LB, is that the client (DoS attacker) will get a 503 from the LB after timeout. So, no gain even if your timeouts are reasonable.
It'd be great if you could return a custom response from nginx that would tell the LB to drop the request -- or you could move the exploit-detection logic to the LB instead of nginx, and the LB could do its own 444 equivalent.
https://wiki.nftables.org/wiki-nftables/index.php/Moving_fro...
To be fair: I just downloaded a ipfw setup and didn't give it much thought. I spend several weeks hand crafting several ipchains scripts. I spent ages with iptables and wrote a rubbish multi WAN effort and eventually ditched it for pfSense for edge. More ages for a host based effort. I also use ufw quite a bit for iptables. I use firewalld for nftables, these days.
I work in IT and it is now probably 25 years that I keep hearing that IPv6 is round the corner. "Adoption" is ~35% but what this means that in 35% of the cases, you can get to a place though IPv6. This does not mean that you must, or do. It is just the capacity.
When a technology takes 25 or so years to be mainstream it means that there is a problem somewhere ("too complicated", ...) or that there is no problem in the forst place ("iptables work fine for me", "I NAT my 10.x network", ...)
Trusted combo: Fail2Ban + 7G firewall
What I don't understand is the second part -- blocking those hosts. Seems pointless now that you've de-noised your logs. They're still sending packets. Saves thousands of bytes on outbound?
What about all the scan-spam on sites WITH host headers? Whatevs.
For scan-spam that does hit your "real" site, it's a bit more tricky as there absolutely will be false positives. You can grep for all 401/403's and add them to the list, but that will sooner or later hit a real user. So it's much more specific to the application you're hosting, where this works for just about any site. The other nice thing is that even when they scan your "real" site, they'll often hit the default host via IP scans at the same time, so you can still manage to ban them.
It's not perfect, but it's good enough :)
When I started banning IPs that send "HELO <myhostname>" for 24 hours, I cut the number of fake login/registration attempts on a bunch of my web-based projects by ~50%.
It works the other way, too. Temporary bans on hosts that try to access /wp-admin (I don't run Wordpress anywhere) cut my email spam significantly.
(Some day, I'll get around to implementing a real reputation tracking system, with exponential ban lengths.)
Do note that doing this kind of thing can block people on Tor, because Tor is used for attacks quite often, also.
You can add something like this to your robots.txt:
User-agent: *
Disallow: /some/unguessible/url
I had some problems to get community support and it seems that activity around firehol is fading away and I am not sure whether this is because this is a complete, finished product, or because it is abandoned.