I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work?
I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else.
And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.
I've used this story a couple of times before on HN, and if it hadn't happened to mother's coworker, I would have thought it a preposterous joke. Mom's colleague was buying something during her lunch break, and had her card rejected. Got the same result in another store, so obviously thought her card had just broken.
Went to her local bank branch to ask what's wrong. The clerk took the card, looked up from her screen and against the evidence in front of eyes declared: "You are dead." Once that kind of flag starts to make its way through our computerised systems, it is very difficult to reverse. It took several weeks to get the error corrected, and in the meanwhile other systems that trusted the bank as a source of truth had picked up the information and started to apply it to their accounts.
Eventually it came to light that a full namesake's estate had had their accounts finally closed, and the clerk doing that had done only a name search when applying the "account terminated, owner deceased" flag to the accounts. Bank never took responsibility for their process failure. They fought for a good time to even have to accept that there had been an error in the first place.
Which incidentally is not entirely unlike the situation in this perversion of justice.
So true. I went through the Wikipedia article, and couldn't escape this feeling:
1) Some errors were made (faulty IT system).
2) Handling of said errors expands enormously into various reports, IT vendor denying their system is faulty, legal proceedings, criminal convictions "tainted" & overturned, etc.
To the point where original errors almost seem irrelevant within the bigger frame of subsequent events.
Imho all the mistakes were made in the processes concerning 2), and 1) is a tiny side-show at most. IT systems can fail, period. What matters is how that's handled.
I transferred 20k from savings to checking. During the process I think what happened is the web banking page froze and allowed me to hold the enter key on confirm that repeated several times. I ended up with 200k in my account that came as a negative 180k balance to my saving account sent to my checking account. (values rounded for story simplicity)
This meant that not only did their online portal not have controls to make sure that internal money transfers are confirmed before processing. They also had no controls to make sure that when an internal transfer is done that the money is there to be transferred.
I waited a couple days expecting a call. Nope. At the end of week I call them. Get transferred around a bunch. The whole time everyone is saying everything is good nothing is wrong, there is no problem. I 100% certain every single one of them was lying through their teeth while they go the FBI on the phone because of the automatic assumption that the computers are right and I am automatically a criminal. There is no way all those people saw a -200k balance in a saving account and thought it was fine. (BTW that means it is standard bank policy to lie to a customer when something is wrong)
So what happened. They immediately zeroed my accounts and froze all my bank activity for three months while they "investigated" Then they only returned 15K of the original 20k that I moved so they effectively stole 5k from me. I spend another 6 months trying to get that money back and did not succeed.
I would have been screwed if I had not been born poor and was familiar with the banking system corruption. I have my income,spending,saving and investing split among different banks and banks accounts in case of something like this ever happened.
The bank initials are WF.
"Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. ..."
My spouse did run into some nonsense some years ago due to her checking and savings accounts being at the same bank and automatically linked and mixed-up. Banks are always offering you super convenient awesome features with "protection" and "connection" and just PLEASE NO - that's how one mistake, that was that bank's in the first place, takes down all your available funds. You'll probably get most back, eventually, but do you really want to live with that for some weeks/months ...
https://en.wikipedia.org/wiki/Systemantics
now this is depressing. I hate everything and I live in a constant 'everything is shit' state of mind, but this book hates things way more than me.
We’ve had millions of years of evolution to prune out problems, but in many cases the problems were papered over by a kludge that introduced other problems, which were then kludged, etc. No wonder the complex systems we build are such a mess. They’re a reflection of ourselves.
Imagine something with 200 billion individual parts with individual functions (and that’s just the cpu) that is presumed to work perfectly flawlessly billions of times a second for a decade.
By contrast, the crankshaft on your car goes through a bout 1 billion revolutions before it is worn out. Your heart might beat 4 billion times before you die, if you are really lucky.
The precision and reliability we take for granted from modern computing hardware is dumbfounding.
Wait till you uncover what the SW runs on :). From math, processor/memory, rtl design, verification, tools that generate the circuits to the fabs that create the ICs.
And when I submit this comment it appears for others to view.
It is a miracle this entire structure works. It never ceases to amaze me.
What must not to fail has very rigorous standards, e.g. Nasa, aerospace, and medical devices.
Everything is built like an onion of failsafes.
Beyond that too, I think it's just important for people to realize that most things just don't matter that much. Everything we do eventually has to reach human scale and at human scale, there's so much room for slop and correction that most failures don't matter. Missing a paycheck because the payroll computer dropped your row is awful, but society can be built around allowing people to recover from that. Late payment forgiveness, forbearance, procedures around cutting a new check early.
I think the biggest danger about the "presumption that computers are working correctly" is in it's ability to erode our understand of the fact that "to err is human". It's too easy if you assume the machines are perfect to assume that any errors are therefore malicious, and so it's easy to go from a system where you have "oops my tax software messed up, no big deal I'll file an amended return" to "the only way for a mistake like this to happen is to intentionally put in bad data, so you must be guilty of tax fraud". Or even worse "the computer won't let me do it". I understand you want computers to have check in place that prevent the overriding of procedure and you don't want to give your low level employees the ability to override procedure at will. I also think that EVERY system should be built such that it is possible with a certain combination of people on site at the point of contact to override ANY check in the system to make the state of the system match what it should be. My specific example in this case was a problem with a prescription at the pharmacy. The script had been partially filled because they had stocking problems, but instead of going into the "partial" state, it was closed as filled completely. The tech knew what had happened and knew we still were owed the rest of the script. The pharmacist knew what had happened and agreed we were owed the rest of the script. The store manager agreed. And yet no one could change the state of the system. The only option was to get a new script from the doctor for a new fill entirely. That's the danger of assuming everything is working correctly all the time.
That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits.