It seems routine to see a bunch of browser User-Agents from the same IP

IP addresses vary in scale in terms of the number of devices behind them. There are many, many reasons for this including corporate gateways, mobile networks, CGNAT, VPNs, public wifi and yes, even home NATs.

There's no "normal" UA cardinality count that you can use for filtering. However, there may be other traffic characteristics that are useful; the shape of the distribution, for instance.

We sell to K12 schools. Depending on the district sometimes they'll have a single IP for an entire district. If not, often a single IP for the building.

This makes things like DoS protection... Interesting as we can regularly have thousands of users legitimately using our stuff from the same IP concurrently.

Some of the Mastodon traffic is because of https://masto.host

The Internet is full of bots. It is full of corporate bots that scrape data for AI. It is full of government surveillance. It is full of North Korean Lazarus hackers. It is also full of tech enthusiasts that have various projects. It is full of people that use various RSS readers.

I often am repulsed that servers reads user agent. It is often not used for compatibility reasons, but just to detect if you are running official browser, or if you are 'privileged user'. This leads, obviously, that some programs needs to use disguise, which would not happen if servers were friendly toward friendly bots. Imho there is a problem with bad-bots, not all bots, that take away your bandwidth, processing power, etc. etc.

If the goal is to have a decentralized web without the Cloudflare protection racket, would proof of work be feasible and/or desirable to deter mass crawling? At the heart of the issue is that sending a http get / is cheaper than generating the response, tilting the economics in favor of the content hoarders.

Or is mass crawling of static content completely fine and we should just be better at caching?

While I suspect this may be old news to a lot of the HN crowd, this article interesting information that a lot of people may not know.

I ran a website years ago that was targeted towards college students and would pretty consistently see many (hundreds) different UAs under the same IPv4 address due to NAT. Let alone proxies, VPNs, etc.

Yet every once and a while someone will suggest the idea of rate limiting based on IP so it's definitely not universal knowledge even among developers how common it is for multiple users to share the same IPv4.

CGNAT :)

To anyone that has ever tried to parse their http server's access logs, this is something that they realize and come to the same conclusion. This is one of the reasons why third party statistics like GA exist. If this was a blog from someone that this was their very first attempt ever at parsing logs, this might seem more reasonable. However, this person claims they know parsing access logs is frustrating and yet claim this was a novel/clever idea which does not compute.

Just a simple web search on issues parsing access logs would show this is not a new idea nor would it be a successful one.

My browser is configured to send fake referrer and user agent strings. I tried not sending a UA header at all but it broke too many websites.

Has anyone had success using things like the rsync or bgp feeds from Spamhaus to drop this traffic?

TLS fingerprinting can spot things like requests made with python requests easily.

Crawlers.