Exciting to see this on HN. I think very soon agents like Skyvern will account for the vast, vast majority of web traffic.
There's no reason for somebody to create a website, pay for resources, and hope for some sort of revenue if their visitors are mostly AI.
So why bother creating a UI? Instead it would make more sense to close the website and offer the same information as a paid API service.
Any sort of website that needs to validate human visitors will be plastered with DRM. Rendering these web browsing LLMs useless. And good riddance as well.
Using an LLM to browse the internet feels like a huge waste of resources.
Instead it would make more sense to have a wikipedia-like for AIs to crawl via embeddings.
This kind of pattern makes it so you can serve both users and agents with a single interface
So what? I bot protect my site, redirecting the AI to a minimalistic part that most likely expects some sort of value given?
People will just breach this trust, like OP and abuse tools like Selenium (as they always have) to imitate being a human.
Perhaps a similar thing as robots.txt is in order (agents.txt?)
And yeah, that sucks for content farms but putting up content and getting nothing in return is already how ad blockers work and it hasn't destroyed the them. I seriously doubt that AI traffic will even put a dent 1/1000th of the traffic loss of Google snippets.
I still have fear that the real internet has already split from what I see and I was left behind.