Comment by janmo | Hacker News Reader

Top New Best Ask Show Jobs

Comment by janmo | Hacker News Reader

Parent Full thread

His robots.txt explicitly allows bots including LLM bots to scrape his site

The LLM scraper bots ignore robots.txt

Permission to scrape is not permission to DDoS.

Reply on news.ycombinator.com