NoML Open Letter

That's the problem with this kind of complaining generally. Some authors (a small vocal group anyway) want their work to be indexed, searchable, publicly available with no friction, but they somehow want to control how it's used. Anyone can make a locked down site that prohibits ML trainings (which would be futile against a determined adversary but would prevent crawling), but they don't want that, they want all the benefits that come from the public internet. Releasing something publicly means giving up some control, that's reality.

The only problem is that the other side already ignores robots.txt. What is the difference in processing and indexing something to be returned as a full original result vs being returned in a permutation? The point of robots.txt is to control indexing, something the LLM/ML businesses are completely ignoring already because they’re “different” and don’t have to play by the rules because AGI.