Is it really that hard to make utility of a crawler visit to ones page negative? Hidden links with boatloads of garbage, generated nonsense and so on ...
Seems like the only viable method how to fight back to me.
Markov chains are almost free compared to LLM generation, and the content, from a machine point of view, couldn't simply be ruled out as non-human unless you had a smaller model in the background pruning out that sort of thing.
They don't need to fight "AI" and prevent it, they only need to make sure their work isn't stolen at individual level, and there are tons of options they can adapt.
Not exclusively, no. As we march forward I'd expect:
- the number of web pages generated by AI will outpace the number generated by humans, probably on the order of a few magnitudes, meaning that human-generated content will drown in a sea of machine generated content. Furthermore, ad-supported web sites will all be AI generated, cutting off advertising funding for human-generated sites
- this will make web pages a very poor source of information to scrape overall, since it'll mostly be LLM output
- sophisticated AI builders will start signing licensing deals with content creators that gives them exclusive access for their AI. Think: Medical and legal journals, large archives of historical works, stock market historical data, technical manuals, etc. This content isn't very consumer friendly as-is, but could be used to generate consumer friendly content that is technically more accurate.
> It's weird how you paint that it's inevitable, just to tell them "tough luck"? They don't need to fight "AI" they only need to make sure their work isn't stolen, and there are many options they can adapt to.
I think you're suggesting that people can continue publishing their content on publicly accessible web sites and that they're somehow able to detect that their site is being visited by an AI web scraper, and in those cases they feed "garbage" to the scraper. I'm suggesting that this will be a loosing battle, that trying to detect bots already is a forever-war that can't be won, while at the same time consumer behavior will change such that publishing pages on public web sites won't be an effective way to reach an audience because of the mountains of AI-generated content.