It looks like for it to work like this the doc would need to be divisible into obvious units that could be retrieved by some key/terms, but in our case it was much looser than that. There were no fixed rules (otherwise we'd have written business logic), it was guidance that really assumed the whole doc as context, and human judgement of the relative weightings of parts of the doc.
The failsafes based on confidence levels is an interesting idea though, that is very possible. I suspect we'd have started out with manually reviewing all decisions and slowly backed it off as the prompts got better.