This is more "project this result onto a context -specific human like space of answers" than "compute it all in LLM"
This is different from say code generation where, to first approximation at least, automated tools (compilers, unit tests) can do this cheaply.
I don't see an equivalent approach that can be applied to LLMs.
I said to him "Would you really take a report into a meeting without knowing 100% what it was going to say on it beforehand? Even in the summary, do you want to risk presenting some wildly inaccurate LLM output to your boss and peers?"
Essentially: Accuracy is only one thing we want out of software. Replicability is another one. LLMs are terrible at the latter and only "good enough" at the former
Better use cases for LLMs are either where:
a) There is a non-judging human in the loop who is using the LLM as a tool, and accepts it's errors/limitations. It's a bit like asking a co-worker for advice - they might have something useful on occasion, but you're not just going to take what they say unchecked and put it on the company web site or send it to your boss as accepted truth.
b) You can verify the output of the LLM in someway - maybe easier if it's a structured output of some sort (SQL query perhaps) rather than free-form text.
b) if it was that deterministic, then you could generate it without an LLM.
Other than just playing with them, the only place I've personally found any utility in LLMs is essentially as a kind of search to get a toehold in a subject matter I'm not familiar with.
For the summary thing, I think we eventually settled on a workflow that generates the summary using LLM, then we store the output in our dataset for the report. This also allows the human building the report to modify it if necessary
A failure in this particular space in an observability product we use leads people wildly astray. While the data is correct, the summary is definitely not because it does not have enough understanding of the context.