Hacker News Reader

This is a file for a LLM, not a scraper, so anti-scraping mitigations seem sort of beside the point.

And to try to get them execute bb(5) ;)

claude --plan "let's develop a plan to detect and mitigate tarpits"

Ten minutes later, the ball is back in your court.

Do you think an LLM would be able to generate a solution to a novel problem just like that?

That doesn't match my (albeit limited) experience with these things. They are pretty good at other things, but generally squarely in the real of "already done" things.

Anti-crawler tarpits and related concepts have existed for decades already; LLM training data is only the latest and most popular of web-scraping goals.

Claude is happy and able to provide a laundry list of ways to mitigate the impact of tarpits on your crawler, and politeness / respecting robots.txt is only one of them.