How are you measuring the confidence of the answers? This is one of the biggest challenges I've seen with AI, it provides wrong answers to hard questions, which wastes the user's time.
tl;dr we define a JSON schema with a few semantic labels that represent a gradient in confidence. On our end we prompt it with certain examples and guidance for when to use each label. This is generally a better approach than e.g. asking an LLM to give a numeric score.
We also have trained embedding-based classifiers as non-LLM heuristics.