Hacker News Reader

Comment by adzicg | Hacker News Reader

sure, then it all falls on its face due to a stupid bug. I'm currently writing a book on cases where bugs caused serious damage or harm to people and it's amazing how many stupid assumptions people make believing that tech is always right. e.g. Georgia Department of Driver Services couldn't handle photo recognition for twins -- their fraud detection systems insisted it was the same person and rejected driving license test applications as potential fraud (same person applying twice under different names). BBC journalists managed to trick HSBC voice authentication (again using twins). Massachusetts Registry of Motor Vehicles falsely identified a person by matching his drivers license photo against an anti-terrorism database and revoked his license, wouldn't even tell him over the phone why. It's just a matter of time before some really serious mess kicks off because a computer wrongly recognised something.

View on HN

But all of this was happening before, and it's still happening all the time: people getting arrested (some even convicted) just because they looked like someone else, bureaucrats refusing to issue documents to people because of some paperwork mistake, etc. My uncle had at one point to prove to police that he's still alive in order to get a new driving license, because someone with the similar ID number died, and they accidentally marked him as deceased. These kinds of errors always existed, and probably will always exist, just that before they were always caused by human factor, and now we have technology related errors too.

I've been the wrong side of a case of mistaken identity by a government department. It took a very long time to get it sorted out, despite the obvious impossibility of the situation their system claimed we were in, and while we aren't talking life and death, the consequences could have been quite serious.

The most frustrating thing was trying to get anyone to acknowledge the possibility of the error in the first place. Even some years ago when this happened, there was so much blind trust in the almighty database that it was basically dumb luck that I eventually managed to find someone willing to take a deeper look without just hanging up the phone or otherwise cutting me off.

The biggest underlying problem, apart from the original error that caused the whole situation, seemed to be that despite my complaints there was no sort of formal case opened to deal with the problem, nor any clear set of records kept, nor any specific individual assigned to resolve the matter. The whole system simply didn't acknowledge the possibility of a mistake or provide any process for correcting it.

At least in my case the problem was obvious to me. With modern systems making automated decisions about anything from mortgage applications to who goes on a secret no-fly list, there seems to be a very real danger of innocent people suffering very serious problems as a result of these errors but having no way to even identify the cause of the problem to start trying to get it fixed.

agreed, but now it's going to happen at scale.

eg. in australia, they launched the derided 'robo-debt' system that sent people notices about potential benefit fraud charges, and the harassment of people caused senate investigations in several states. turns out the new system is as correct/wrong as the manual process (20% error rate), but it's now able to process several order magnitude more cases, so effectively floods downstream workflows (claims, complaints, courts) that were not ready for such an increase in demand.

might be just me, but that process sounds seriously flawed with a 20% error rate, automatic or not...

As they've said for some time now, "To err is human, but to really foul things up you need a computer."

"Computers are great! They do the errors much faster and with greater precision!"

Tech isn't always right, just like human. However, if tech can get close to human level accuracy, it is worth pursuing.

Reply on news.ycombinator.com