Hacker News Reader

One way that Cloudflare is gatekeeping is by declaring which bots are AI Bots. Common Crawl's CCBot is used for a lot of stuff -- it's an archive, there are more than 10,000 research papers citing common crawl, mostly not AI -- but Cloudflare deems CCBot to be an "AI Bot", and I suspect most website owners don't have any idea what the list of AI Bots is and how they were chosen.

View on HN

would be an obvious loophole if you could just use CC instead of paying cloudflare no?

It's a similar loophole as public libraries. When I was a kid, I read thousands of books from the library, without paying anyone anything.

But as for the crawl loophole: CCBot obeys robots.txt, and CCBot also preserves all robots.txt and REPL signals so that downstream users can find out if a website intended to block them at crawl time.

Reply on news.ycombinator.com