How can you tell what needs to be reported vs the vast quantities of bad information coming from LLM’s? Beyond that how exactly do you report it?
Internal audit teams, CI, other models. There are probably lots of systems and muscles we'll develop for this.
Although they don't necessarily look at any of the reports.
They don't look at your chats unless you report them either. The equivalent would be an API to report a problem with a response.
But IIRC Anthropic has never used their user feedback at all.
Meanwhile essentially 100% of lengthy LLM responses contain errors, so reporting any error is essentially the same thing as doing nothing.