The expectation should be on the LLM to admit they don’t know the answer rather than blame devs for not allowing crawling.

"How do you expect the gangsters to protect your business if you don't pay them?"

In many, if not most cases, the producers of this information never asked for LLMs to ingest it.