Hacker News Reader

I made this, let me know if you have questions or feedback.

Thanks for the work on this!

I automated my site's robots.txt[0] by scraping your site. It would be extra nice if darkvisitor.com exposed a plain text version or JSON representation of the list.

[0] https://tbeseda.com/blog/automating-my-robots-txt-to-block-a...

That was one of the points of the new repo. A plain text version of the file is https://raw.githubusercontent.com/ai-robots-txt/ai.robots.tx...

The other point was to make this community maintained rather than rely on one source to provide all the inputs.

Definitely! And I'll likely use that raw URL in my tooling going forward. So thanks for starting the repo.

I'm thinking it also helps to bring up a feature request on the source material so we can all limit the drift. ie. if darkvisitors.com had a sort of plain text API, your repo could check for new entries via GH Actions and create issues or even PRs.

Reply on news.ycombinator.com