Ask HN: How can I trace what user queries make AI bots crawl my site?

4 pointsATechGuy2 comments

I've noticed several AI bots crawling my website. Is there a way to figure out which specific user queries caused the AI to access my website?

View on HN

It might be present in your web server logs, with the caveat that you may not have them or have access to them and the bots may not pass the HTTP_REFERER header and the logs may not record it.

It might be present in client-side logs, with the caveat the crawler might block tracker URLs or not open a page in a way that executes their scripts.

Either of these may reveal what the first-page was a crawler reached, or the pages they directly-access without navigating your site.

You might be able to correlate timing for when a "user query" occurs and subsequent scraping.

Unless these AI bots choose to pass query data in these UTM parameters, i'd say there's no way to obtain such information, except for you to go into an ai and try to trigger the ai pulling searches that contains your website urls.

Reply on news.ycombinator.com