Yes, we can't block models from scraping the public repos and train on them, but *we* won't use your code (neither public nor private) to train models on them. We really only want to host your code and do that well.
FYI, there is the nuke option, which is generating endless nonsense pages as a form of “bot sink” [1] that ends the scraping relatively quickly, but IIRC that also tanks your search rankings, since it likely affects benign crawlers too – not something you’d necessarily want to happen to a new domain, unless you really need to protect server resources against aggressive hostile crawlers like SourceHut and so many others had to combat.
[1] https://www.toxsec.com/p/ai-tar-pits-are-drowning-llm-scrape...
A developer would be the one using the LLM to evaluate page response text and scraping what they need.
LLMs cannot “crawl pages” or “scrape” or do anything - it’s an offline archive with a few convenience methods.
Being a bit needlessly pedantic here. I think it's well understood in these circles that such models are being driven by agents, which absolutely can crawl and scrape. The only thing a human needs to do is write rules for the agent to follow.
It’s also not true.
An LLM cannot visit the internet at all.
There is no “agent” - that is a marketing term. It’s software and LLM that’s it.
Wouldn’t matter in court anyway or anywhere you put it to the test even if you called it bits.
Point is dodging accountability by saying some “AI did it” when it’s not reflective of reality.
LLMs are read only and have no access to the internet :)