I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.
his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.
You may have more luck with a hybrid approach, using LLMs for language understanding and computers for the counting. For example, ask them to write a short, one-line description of every instance where something happens, and then use a traditional program to count the lines.
To be clear: I don't trust it to provide an accurate summary; it's not meant to replace reading the documents. But it may help to find relevant parts of a document later, or get a reasonable overview of documents before starting a manual review. I expect (but will have to see) if it's better than just opening up a random PDF and starting there.
I recently had a case with over 24k pages of records, some of which were in PDFs that were thousands of pages long. The ability to do semantic search rather than keyword search was useful for when I said to myself, "Didn't I see something about that before? Where was that?"