The danger I see here, is that if you ask an LLM to explain the thought processes, it will never say “I don’t know”. It will instead describe some thought processes associated with coffee grinders. It may say something like “this grinder has fine grain controls that allow customizing the size of grind.” …which that particular grinder doesn’t have at all…but that’s a thing people write about when choosing grinders. The frustration is that 90% of the answer will be accurate, but somewhere in all the sentences is a hallucination, treated with the exact same authority as the rest of the answer.
It’s very difficult to QA that type of error.