Hacker News Reader

Comment by nemothekid | Hacker News Reader

One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert, you are immediately blasted with ai crawlers.

I put a project online - it was online for a month, and the second I added an SSL cert it went from 0 traffic to 1000 requests/min.

View on HN

> One funny thing I've discovered as a result of certificate transparency logs is that the second your host gets given an SSL cert

I've been thinking of using wildcard certs for Caddy in regards to this.

and then what? serve your app under some obscure / customer unfriendly subdomain?

Even if you use a common subdomain, anecdotally I get orders of magnitude less bot traffic than not using a wildcard cert.

Make a new certificate, let crawlers blast you and add those IPs to a block list.

these old network security techniques don't really work anymore. the common bots are at known IP ranges, the problem bots are all on datacenter + residential proxies.

Why would blocking those be a problem?

there are 150M+ of them and you'll be taking out a lot of human users with it

modern blocking is behaviour / heuristic based

There are 150 million bots all using residential IP addresses?

because you are blocking all of Comcast, Verizon, T-Mobile, British Telecom, ....

at the end you have blocked every network with human visitors and only datacenter IPs can access your site.

The proxies rotate IP every day, so you either have ineffective blocking or you block the whole network.

My site is not for americans so I don't care about blocking american isps

You think they only use American networks?

In my experience, these aren't the crawlers from legit companies, so they have infinite IPs via residential botnets/proxies.

edit: 'nikcub beat me to it by 30 seconds :)

Today AI crawlers, years ago vulnerability scanners from Russia or China.

Either way! People monitor cert registries for targets.

Reply on news.ycombinator.com