I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.
I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.
I would be positively surprised.
I was indeed wondering how constraints on model interpretation forces governments to stay away from NN and more complex approaches.
I have worked for a fraud-prevention company for a few years, and the financial nature of the decisions involved led to reward model interpretation (sometime at accuracy's expense) in order not to create friction with clients. Definitely not the best not being able to explain why your model performs poorly or what actions are needed to solve a problem.
Now in practice .... Our IRS has had a discriminatory way of working in place where no-one involved had access to their own files (GDPR?!) and even judges were reluctant to order transparency in the following proceedings giving the IRS the benefit of the doubt. Ultimately this became a politican scandal and will take years to resolve. And in the paragraph above I would have argued that the basics considering any case should be easily available in as much as 8 weeks for every citizen...
I think there's no other way around this: ultimately, everything that has legal implication will be potentially subject to dispute. We're not yet in the dystopian environment where disputes can be handled by a model :)