in my observation: when humans are automated out of a process due to the human element being inconvenient, the perceived efficiency gains are often because wronged individuals have less recourse in the automated system.
> In another example, a Vietnamese national was initially given the benefit of the doubt at the first triage that took place in the waiting area. The CIO and social worker commented on his “soft face”, which they said was consistent with his claimed age of 17. However, his “developed shoulders” and “huge hands” cast doubt for them, as did a “tiny bit of stubble” that they noticed when they asked him to raise his chin. The CIO and social worker told inspectors afterwards that Vietnamese young people were typically difficult to assess because they “did not have the same ageing process”, and “did not show signs of ageing”. When asked where the evidence for this was, they said that it was knowledge gained through their own experience. The social worker said, “It is just genetics”, but was unable to support this with evidence.
If I had to choose between being judged by an AI model and being judged based on ad hoc stereotypes of what my race's shoulders and hands typically look like, I'd definitely pick the AI.
It's a problem when people use this kind of system to circumvent the question of "do we have to make this judgment at all". We shouldn't, for example, predict from someone's photo how likely they are to commit a crime, so we're rightly skeptical of people who try to argue about system X or system Y might better predict it.
But as the source article covers, the UK's asylum laws require it to make this age judgment, because child migrants are entitled to special programs separate from adult migrants on account of their vulnerable status.
It's true that the model might develop the same strange belief that large hands prove a Vietnamese person is not 17, if the training data is biased in that way. There's no perfect solution.