Hacker News Reader

Comment by atomman747 | Hacker News Reader

I get that the author “doesn’t care anymore”, but I saw Google, ripe.net, and semrush in the banned IPs.

Of these, I certainly wouldn’t ban Google, and probably not the others, if I wanted others to see it and talk about it.

Even if your content were being scraped for some rando’s AI bot, why have a public site if you don’t expect your site to be used?

Turning the lights off on the motel sign when you want people to find it is not a good way to invite people in.

View on HN

Semrush misbehaved so badly for such a long time with various levels of incompetence that I have special notes in my robots.txt files going back at least 8 years and eventually got someone in legal to pay attention. I see zero value to me or any potential users of mine in letting an 'SEO' firm lumpishly trample all over my site barging out real visitors. And some of Semrush's competitors were just as bad.

The current round of AI nonsense also very poor. Again had to send legal notes to investor relations and PR depts in at least one well-known case, as well as all the technical measures, to restore some sort of decorum.

My experience with them has been the same. On one of my employer's websites, the top three bots were Google, Bytedance and Semrush. It is a small website for a niche audience, not even in English, and changes very infrequently (like once or twice a quarter). That did not stop these three bots from hammering the site, every second

We were getting hammered on nofollow links. Someone made a change to stop putting the nofollow in and using JavaScript to traverse on interaction, but that person left sometime after the toggle went in and we discovered it had never been turned on much later, when we were rearranging our ingress load balancer and needed to tweak how we handled bot throttling. That was at least 10% of the requests during peak traffic and extended the duration of the crawls by a lot. So dumb.

Because the bot requests are consuming significant amounts of bandwidth, memory, CPU, and disk space. Like the intro says, it's just rude, and there's no reason to serve traffic to harvesters like that.

Google also runs an AI scraper, which might be what you saw represented there?

From the article it's sure starting to seem like people across the internet are just starting to realize what happens when you don't have just 3-4 search engines responsible for crawling for data anymore. When data becomes truly democratized, its access increases dramatically, and we can either adjust or shelter ourselves while the world moves on without us.

Did Google never ever scrape individual commits from Gitea?

> When data becomes truly democratized...

That is not at all what is happening.

I know, we're locking everything down behind WAFs and repeating captchas so only attested identities can get access in the end.

Yep, everyone is building their own little walled gardens instead of adapting.

Friend, this IS them adapting.

More like maladapting.

Can you please provide an explanation of what you consider 'adapting' to be?

What's described in the article did my personal definition pretty well.

Ok, and, what your solution?

You haven't started educating people in the last ten years that it's just not acceptable to use Google any more ?

Especially when this happens ?

Google is using AI to censor independent websites like mine

https://news.ycombinator.com/item?id=44124820

Sure sounds like we've reached the point where it's more of a liability !

There are also bad actors who pretend to be the Google scraper. Google once upon a time had a reputation for respectfully scrapping, but if he's getting the traffic he needs with or without the Googlebot, why should he care?

They had those stupid tags telling them far more anyway. Nevermind what it was doing to time to first paint/interactive.

Reply on news.ycombinator.com