That was my first thought as well. An LLM seems more like a search engine for the library only instead of indexing through topics it tries to find understandable non-sense in the sea of non-sense.

The difficulty with them is that they can only find things similar to what they've been trained on.

Lacking a curated index it's not going to find the information you need or want; it's going to find things that seem most like what others have seen or wanted.

I thought of https://en.wikipedia.org/wiki/On_Exactitude_in_Science and its relation to language, which is discussed by Baudrillard's Simulacra and Simulation. Large Language Models are the efforts of mapmakers to encapsulate reality in a way that is ultimately futile.

Interesting how this work received an homage in The Name of the Rose, up to the name of the librarian, Jorge de Burgos. (I hated the way they pronounced the name in the movie, /'iorge/ instead of the correct Spanish way /'xorxe/)