But banks, at least in my country (central EU), don't have to explain their reasons for rejecting a mortgage application. So why would their automated systems have to?
There is so called three line system -- operational line does the actual thing (approves or rejects the mortgage), the second line gives the operational line the manual to do so the right way, the internal audit should keep an eye that whatever the first line is doing is actually what the policy says they should be doing.
It's entirely plausable that operational line is an actual LLM which is trained on a policy that the compliance department drafted and the audit department occasionally checks the outputs of the modal against the policy.
But at this point it's much easier to use LLM to write deterministic function in your favorite lisp based on the policy and run that to make decisions.
[1] https://en.wikipedia.org/wiki/Equal_Credit_Opportunity_Act#R...
[2] https://www.nolo.com/legal-encyclopedia/do-lenders-have-to-t...