Hacker News Reader

> When it comes to the #OpenWeb, I am ride or die.

Isnt the whole idea of #OpenWeb open access to information? How does blocking using robots.txt in line with that?

Because parasitic entities tend to misbehave. There's always been other search engines, Google is in slow decline (or accelerated, depending on your viewpoint) so blocking them because they slop users into oblivion. Sometimes you have to say: This his how far I was willing to let this go and now it's over.

heres the rub, whos to say in the future or even now that google is really respecting the robot.txt in the first place. I know for a fact that my sites are still getting hit by AI bots but I'm "blocking" them via cloudflare. Way too many IP's originating from data center ISP. Just saying respecting the robot.txt maybe just PR lip service.

Reply on news.ycombinator.com