While I love to see openai get scammed I don't think it will stop there. How cheap and useful do you think Kagi or other search engines can stay with this racket? How will Internet Archive operate?
Everyone should remember, limitations of technology is not meant to define society. Instead, we build edge cases into technology to better match society’s general expectations.
A website owner saying “yes normal humans, no bad bots, EXCEPT good bots” is totally fine.
If websites owners truly wanted it, it would be a 'do thing to opt in' and everyone would rush to that.
Now I do think this kind of thing is good for many reasons, but I also see many reasons this can be problematic (that I did not consider the first time I read about it).
I myself would prefer an option to throttle the bots, and give them a 'you can spider at 2am-5am once per month access' via robots.txt, header or something..
you come more than twice in a month and get blocked or pay for access to static version hosted on other server / cdn..
best of both worlds without some of the negative issues.
Otherwise it's a play that helps cloudflare more than anyone else, and hurts more than [open][other][AI] - etc. imho.
Presumably increasingly less and less effectively, at least if they continue honoring robots.txt and don't implement scraping protection bypass mechanisms.
https://www.theverge.com/news/757538/reddit-internet-archive...
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Don't forget that cloudflare provides service to the very botnets and flooders/booters they purport to protect against.
Would that be triple-dipping ? Or do we have a special term for this specific behavior ?
and where is the evidence???
Users are paying for a service that was costed 5-10 years ago based on human web traffic.
Now AI crawlers are a new source of huge traffic volume and CF is figuring out how to cover costs or profit from that load.
Markets change and so should cost structures.