At the token layer an LLM can make things up, but not as part of a structured pipeline that validates an invariant that all suggestions are valid entities in the database.

Can google search hallucinate webpages?