With complicated algorithms it may not be obvious, but then you can try to poke the black box with thousands of sample cases to see how it reacts.
Compare this with humans, who often don’t even know themselves what their decisions are really based on.
You'd have to have access to it. This is highly unlikely. If testing as a black box, could they identify a test and change its behavior, like VW emissions tests?
Only the company itself is actually in practice going to be testing their system but they only do so with a view to their own interests.
I think the GDPR legislation already takes small steps to this direction.
Like, even if the dismissal was after a warning, because of rules well known in advance and with an accurate explanation, GDPR would still prohibit making this decision solely through automated processing, there has to be a place for human intervention.
Article 22 even acknowledges this to some extent, making an explicit exception for people who have some other kind of contract (including employment) and for which the law has defined suitable safeguards. U.S. employment legislation is indeed not a sufficient safeguard, but the fact that it insufficiently protects privacy is somewhat of an understatement.
Shame and empathy are built in because we really do work better in at least small groups of up to perhaps a few hundred people.
> All the while, the potential for discrimination based on race, gender, or disability, especially in hiring and firing software, remains obscured behind the black box of AI.
I would have disagreed 2 years ago but got a new boss last year that seems to make huge business decisions based on a random number generator.