HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by slipknotfan | Hacker News Reader
Parent
Full thread
slipknotfan
·
https://sr.ht/robots.txt
is a robots.txt
View on HN
zionsati
·
Do LLM crawlers adide by this?
gus_massa
·
Ignoring that AI bot will ignore it, ...
Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?
Reply on news.ycombinator.com