Can we force Claude-SearchBot, GoogleBot, OAI-SearchBot, PerplexityBot and GrokBot to give us info about what they are looking for on our website?
What if we stopped treating AI crawlers like static file scrapers and started capturing LLM user intentions from them directly?
Can't we just drop a lightweight on-site agent onto our pages so we can intercept these crawlers and force them to interact with our agent when they arrive.
Instead of blind scraping, we would track what millions of users are probing LLMs for in real time.
What do you think about this approach? Is that a ridiculous idea?