Hacker News Reader

Hacker News

Comment by Avamander | Hacker News Reader

Full thread

Avamander

It's annoying for sure. I deal with abuse at a large scale.

I'd recommend:

- Rate-limit everything, absolutely everything. Set sane limits.

- Rate-limit POST requests harder. Preferably dynamically based on geoip.

- Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount.

- Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break.

- Require SNI. Do not reply without SNI (nginx has 444 return code for that). Ban IP's on first hit that connect without. There's no legitimate use and you'll also disappear from places like Shodan.

- If you can, require HTTP/2.0. Bots break.

- Ban IP's listed on StopForumSpam, ban destination e-mail addresses listed there. If possible also contribute back to SFS and AbuseIPDB.

- Collect JA3 hashes, figure out malicious ones, ban IPs that use those hashes. This blocks a lot of shit trivially because targeting tools instead of behaviour is accurate.

View on HN

> - Rate-limit everything, absolutely everything. Set sane limits.

This breaks when multiple users are behind the same IP. I've seen services fail even in classroom, because the prof did something and a few tens of students followed (captchas everywhere).

A "sane limit" wouldn't be "one person one IP", such a global limit should rather stop one IP (even if it's a nasty CGNAT) from having a negative impact on the entire service.

If such a limit would hinder classroom usage, but that's your target audience then other solutions should be found, fairly logical.

(Author)

I do in fact rate-limit everything, it is good advice, but the way you implement rate-limiting allows for traffic bursts. It's basically a reverse leaky bucket, where you start out with N allowed requests, which gets depleted for each request, and refilled slowly over time.

Search traffic is fairly bursty, people do a few requests where they tweak the query and then they go go away.

Off-topic, but isn't that a normal (non-reverse) leaky bucket? When the bucket gets full the rate limiting engages. An empty bucket allows for a burst without getting full. It slowly leaks over time at a rate that allows a normal amount of traffic without filling up.

Hmm, yeah, that's actually true now that I think about it.

To me, it's a bucket that's being filled at constant rate from a tap until it's full, and the traffic requires taking some water from the bucket. If there's no water, the trafic has to be dropped or wait in a queue.

Basicaly, you can look at it either way.

I like it. I guess it's one of those things where it depends on the example used when you learned about it? For me it was some Nginx guide on rate limiting and I think they described it in the way I see it.

For sure! Our site is B2B ecommerce, and any sizeable customer has all their users coming to us from a single NAT or proxy. For major customers it's likely that there are several of their employees using our system at any given time.

The answer needs a whole lot more finesse than this.

Yup. This happened to us when we had rate limiting turned on our sites and ran off-site events at hotels, for example - then the hotel's IP got temp banned and our sales engineers would complain, rightfully so.

Black hats always find ways around rate limiting, and that's why they are more prevalent than actual users. People can literally run click farms with cheap 4g cell phones that artificially pump anything they want without consequences, while authentic posters that simply run 2 necessary accounts are penalized if they post regularly.

The only real way to properly police Internet communities is to keep them smaller so that botting is more obvious, and to involve carefully managed moderation. Reddit tried this, but also lost track of the human factors involved and now moderators collect side money and promote their own posts artificially.

The main problems facilitating the surge in bots are shammy creator funds and all the other measures sites take to boost their profit and market dominance. They have grown far too big and can no longer effectively manage their user bases effectively. Things weren't meant to be this way at all, the excessive quest for market dominance and profit has thoroughly corrupted freedom of info online in business, now many users are also following the same road map.

> Rate-limit login and comment POST requests even harder. Ban IPs that exceed the amount

Don't ban IPs. Or if you do, let the ban expire relatively quickly (days/weeks, not months/years).

Ideally you'd keep track of repeat offenders and decide the length based on that.

I ban IPs from small data centers all the time. For my purposes there is no need to support traffic from small hosting providers that are everywhere all over the world. I do not tend to ban the IPs of commercial ISPs that provide service to end users.

You will probably ban a lot of VPN users as collateral damage. VPN providers often use these small and relatively cheap providers for their endpoints.

You may be fine with banning those VPN users, or even want that - lots of bots will try to hide behind "legitimate" VPNs - but one has to be aware of this consequence at least, especially considering that more and more people seem to use them - probably also thanks to the aggressive "sponsoring" certain providers such as ExpressVPN do on e.g. a wide variety youtube videos.

It depends on what you are trying to protect I suppose. Banning OVH IPs (and others) cleared up a lot of issues for me. I don't miss them, but sure you might.

Or at least rate limit session cookies. If a person does not have a session cookie, rate limit by IP. If they are authenticated as a unique person have different rate limits and different levels of authentication. HAProxy can do different rate limits by ACL conditions.

Or instead of strictly rate limiting, ask them a question that can't be "looked up" in a table and that requires human thought, philosophy, emotion, ethics. Maybe GPT could eventually adapt to this and in that case fall back to IP rate limiting and grow the set of questions.

"This spam traffic is all from botnets with IPs all over the world. Tens, maybe hundreds of thousands of IPs, each with a relatively modest query rates, so rate limiting does all of bupkis."

Yep, there isn't a silver bullet that curtails all abuse.

Blocking the entire aws/gcp/azure/digital ocean/linode IP ranges will stop 99.999% of malicious bot traffic full stop.

Yes, it would.

It would also stop a not-insignificant number of my customers.

Why do your customers pay Amazon for egress to the internet? Isn't that very expensive?

In other words, make your website unusable for people who have to connect through VPNs or public networks, difficult for anyone without a stable Western broadband connection, and unpleasant for everyone else.

With a bit of work any limits can be fine-tuned usually not to impact actual users behind NATs. Some collateral does happen but that's an unfortunate reality. I'd like you to elaborate on the rest of your comment though.

I agree that there's a "that's life" aspect to collateral/tradeoff.

That said, I sympathize somewhat with the parent. "Done right, negative side effects are minimal" is an uncomforting statement. First, because things are often not implemented correctly. There are a lot of details and tuning that will often fail to materialize in practice. Second, because long tail usability issues can often go overlooked. The abuse->anti-abuse feedback loop is pretty tight. Abuse gets identified and counteracted. The anti-abuse-> UX problems loop tends to be noticeably looser. Often, it's just aggregates (revenue/AUD/etc).

Google search seems to have this issue for most of the regions near me on the VPN I use (Private Internet Access)

Sometimes I just turn it on if I have to fire off a few searches because it'll make me complete a long, tedious captcha for EVERY search

I'm not very familiar with all the workings of HTTP/2.0 - why would it break bots? Assuming no CloudFlare type protection, does it somehow stop someone from using curl to get (non-JS generated) content? Does it thwart someone accessing the site from something like playwright/selenium?

> I'm not very familiar with all the workings of HTTP/2.0 - why would it break bots?

There's a lot of outdated garbage bots out there. Not using HTTP/2.0 is also often the default with various HTTP libraries.

So it just comes down to bot software not being compatible with HTTP 2.0 rather than any sort of HTTP 2.0 specific mechanism/feature?

Yes

> Require TLS. Drop TLSv1.0 and TLSv1.1. Bots certainly break.

So will people who run older computers with older software. But I guess people who don't have money don't matter for commercial websites so screw 'em.

Anything that got a significant update in the past ten to twelve years will support TLS 1.2. The window of systems that would support 1.1 but not 1.2 is pretty small. You have to go all the way back to IE on Windows XP before a lack of 1.2 support becomes an issue.

So, yeah. You're absolutely right. In a lot of cases the loss of revenue from users with severely outdated software will be less than the cost decease of cutting spam and abuse.

This gets back to an old question - to what degree should legacy systems be supported and at what level of expense? There's no one easy answer that works for everyone.

I don't think there's much web you can visit with those browsers anyway. Windows XP with IE and no SNI support, maybe sites from that era without JavaScript would work?

I think you'd be surprised the how recent a browser can be and still lack a client/server cypher overlap once you start whittling down what TLS versions you accept. Just at the start of the pandemic many government sites had to re-enable early TLS because so many people couldn't access their recent TLS only sites.

But yeah, corporate employees aren't going to care about those people. Governments have to. And human persons building personal websites should too.

>Bots break.

Wonder if you could respond in a way to get them to crash, or even better, to hang indefinitely.

If you're reasonably certain that the request is being made by a malicious bot, you could try responding with a ZIP bomb [1] if HTTP 1.1 or newer is being used and the bot accepts compression. It can cause problems for several browser engines.

[1] https://blog.haschek.at/2017/how-to-defend-your-website-with...

I'm sure these would work but I'll probably got banned too just because I often try to poke ip addresses directly. I also often use VPN especially when outside, so I'll definitely got banned.

VPNs tend to be smaller offenders in terms of clients-per-IP than say educational institutions or offices.

> If you can, require HTTP/2.0. Bots break.

Non-bots break as well. I have Firefox configured to use HTTP/1.1 only.

No reason to chase Google's standard-of-the-day, HTTP/1.1 has worked for ages and it will continue to do so for the foreseeable future.

Genuinely curious, why disable HTTP2? Your web browsing must be awfully slow sans multiplexing.

> why disable HTTP2

Because it adds nothing to improve my browsing experience, and reducing the number of protocols supported by my browser from 3 to 1 also reduces the attack surface.

> Your web browsing must be awfully slow sans multiplexing.

And yet it's not slowed down at all. How many different resources must a web page use before it feels slow on a connection pool of keep-alive TCP sockets? Maybe people visit some wild experimental web pages with hundreds of blocking <src> tags that are not bundled/minified?

Either way, my experience is it doesn't slow anything down when I use both websites (forums, resources, youtube, social media) and web apps (banking, maps, food delivery etc).

Perhaps your experience is the same, but it may impose extra load on middleboxes that track TCP flows.

https://github.com/dalf/pyhttp-benchmark/blob/master/results...

HTTP2 is barely any better than http 1, if you want it to make a 1/10 of a second difference you have to be making 100s of requests.

Some old browsers break as well, if it's worth it depends on the website. It's your prerogative to disable an useful feature, you can also disable JavaScript. But there's little reason for a website operator to cater to that unnecessary edge case if it's mostly used for abuse.

JavaScript is mostly used for abuse.

Not in that direction though.

That seems like a strange reason to me. Isn't HTTP/2.0 faster? Isn't it also basically transparent to the end user?

I'm trying to figure out what I would gain by configuring my browser to use HTTP/1.1 only.

If doing something fends off a lot of bots, but also inconveniences a very small number of people who have significantly non-standard or just out-of-date configurations, I'm likely to favour protecting myself from the former over worrying about the latter. To paraphrase Mr Spok: The inconveniences of the me outweigh the inconveniences of the you!

Bear in mind, inconveniencing 4.8% of users, does not map identically.

Instead, you are often dumping 4.8+4.8+4.8 as you add block methods, with some overlap.

To be fair, most of my visitors are not exactly lining up with the expectations of "standard". I get >90% of my [human] traffic from desktop clients, for example.

Sure, but the logic about mitigation does hold true, if you overlap methods.

Eg, your method described in prior post, along with thongs which may lock out VPN or NAT users.

Just something to consider.

FWIW I don't block non-HTTP2, but at some point that 4.8% who can't use it will be much smaller and if you are still turning it off you will be part of an increasingly insignificant population.

It is not dissimilar to measures that will have the side effect of stopping apps working on IE - I wouldn't in DayJob¹ but I'd not think twice on a personal project. I don't have time to care that much about people with very non-standard or very out-of-date setups, if accidentally blocking them is a side effect of a benefit I might feel. Added benefit: blocking them effectively removes potentially difficult users from the support queue!

And to some, that 4.8%+4.8%+… doesn't overlap their target audience enough that losing the few in both categories isn't really a problem in the grand scheme of things.

----

[1] some of our clients still have users using IE11 even though we no longer officially support it² so we still don't want anything other than minor display & performance issues there ATM

[2] of course we will for more money, if someone wants to pay us to address said minor display & performance issues

Reply on news.ycombinator.com