Holding Machine Learning Models Accountable

Good question. Human decisions might even be more difficult to explain fully, so I'm not sure if this regulation makes any sense.

It might in general be possible though to analyze what features where most important to a decision. How exact this needs to be is rather vague to me.

To me it comes down to 2 things:

1) I don't fully understand the "significant legal impact" requirement, but intuitively "Applying for credit/insurance/school/job" is an area where you'd need to be transparent or open to contest. But which product to show based on your purchase history or what your friends liked on <social-site> wouldn't be subject to contest or transparency.

2) The requirements on the explanation are somewhat vague, and what exactly interpretability even is remains an open question. Does a layperson need to understand it or is an expert in the field sufficient? This will probably be iterated and supplemented after some interactions between 'Data Controllers' and the EU. The good news is that this has prompted a lot research on how to understand high dimension/feature models.