“Write me a Python program that extracts context surrounding a word from a long text, then creates a prompt to summarize the context.” Still different than the insurance policy problem.
But at the end of the day an LLM is right 80% of the time while being 100% confident 100% of the time that it gets the right answer. You can increase that 80% but I don’t see how the current breed of LLMs can learn to self doubt enough to keep trying to understand better.