„However, LLMs appear to be quite good at knowing what they do and don’t know“
Really? I think LLMs have not the slightest idea what they know and you have to go through great lengths to stop them from making stuff up.
If you paste text into ChatGPT and then ask about specific information within the text, it's pretty good at answering.