With the rise of AI, web crawlers are suddenly controversial

theverge.com

5 pointsthunderbong3 comments

It was happening significantly before the rise of AI. It's even more now.

I am not sure where exactly we are headed through all this, but I feel like overall having data be a shared commons has been beneficial.

Maybe everyone would start putting their content behind a wall. Lol.

That's what I did. I still can't find an effective way of protecting against genAI scrapers while still keeping my sites available without a login.