Should you blindly trust the summary? No. Should you verify key claims by clicking through to the source? Yes. Is it still incredibly useful as a search tool and productivity booster? Absolutely.
I did check the translations were correct as part of this — while my German isn't great, it was sufficient for this — and it was fine up until reaching a long table about the timeline of events relevant to the subject, at which point it couldn't help but make stuff up.
Still useful, but when you find the limits of their competence, there's no point attempting to cajole them to go further. They'll save you whatever % of the task in effort, now you have to do all the rest; it's a waste of effort to think either carrot or stick will get them to succeed if they can't do it in the first few tries.
> No, it does not.
> It is excellent when just finding something is enough.
How could you say that with high confidence when you admitted it might be useful for others?
1. Get list of sources and their summaries from an LLM.
2. Read through, find a paper who's title and summary seem interesting to you.
3. Follow the LLM's link, usually to an arXiv posting.
4. Read the title and abstract on arXiv. You can now judge the accuracy of the LLM's summary.
It's really easy to tell if the LLM is accurate when it is linking to something which has its own title and summary, which is almost always the case in literature search.