Show HN: robots.nxt – see, control, and convert bot traffic
paynpoint.net
paynpoint.net
robots.nxt provides a free bot traffic analytics dashboard to see how many bots you're getting, what bots they are, what they're costing you, and what content they're crawling.
We provide rules for specific bots, bot categories, and specific content or content categories, and more.
You can modify and combine these, such as rate limiting or access denial to selected areas, and add a pricing model any combination of rules or identities.
For example, if ChatGPT tries to crawl your content pages, you can set robots.nxt so that your server will charge the bot $0.25 per page, but if an indexer crawls your pages there's no charge.
And if a bad bot shows up, you can serve it a 404 or dead, link-less homepage so that the bot thinks there's just nothing there.
If a bot pays for your content, you can include an AI training license.
You can dynamically render robots.txt, explaining what they're allowed to do, while ensuring they can't do anything you don't want. An iron fist in a velvet glove.
You can manage your server responses to, for example, filter any form submission element from any pages served to a bot. You don't have to challenge, because the form never gets served to a bot in the first place.
We can see any attempts to request insecure files or site areas, so you can see if someone is trying an exploit.
Humans will never be the largest source of web traffic ever again. The tools we've built with robots.nxt are an emerging standard for website and web app management in this new bot-dominated internet.
We're starting an open beta today, with all our released capabilities free for the time being. Sign up and tell us what you think!
Want to talk about it? Would be good to connect with someone else interested in this.