The presumption that computers are working correctly
emptycity.substack.com
emptycity.substack.com
When they were eventually confronted with the overwhelming evidence of hundreds of false convictions they doubled down because they lack any sense of morality.
So while it is a sad technical tale it is also an indictment of British Board room culture.
The whole sorry affair has been covered by Private Eye magazine for over ten years.
I don't believe you know what their internal mental states are.
If someone knows they are sending innocent people to prison, benefits from doing so, and continues doing so we can reasonably infer they lack morality.
So given a set of actions you can infer a likely set of mental states — this is also done in the court room. Can you always infer with 100% accuracy and with every colorful detail fleshed out? No. Do you typically need to? No.
And we have we gone through that process to look at the issue from all sides and hear arguments from all parties? No.
You don't need to know intent to fire or punish someone anyway. In this case we can dismiss people for improper action, without demeaning them personally.
I agree that one should be cautious with early accusations or hopping on trains just because they are rolling.
Having rich people face actual material consequences to their life, rather than "punishments" like losing a job or honor, would hopefully reduce corporate malpractice, yes, as they now know that they are not invincible (obligatory note that I am talking about criminal responsibility, not suggesting we do away with LLCs or something)
I think to achieve that it has to be better in expectancy for someone who knows about the problem (and widely believed by that person too) to act sooner to resolve the problem than to try to cover things up, but I don’t really know what rules would help. For example, maybe an alternative thing that would work would be some kind of whistleblower rules like in finance[1].
In the hypothetical where I am stood with my hand on the lever choosing between stopping the scandal sooner and letting executives off entirely or having more wrongful convictions for longer but later throwing the executives in jail, I will choose the former every time, even though it feels unjust.
[1] In finance the whistleblower rules are designed to incentivise people to not be accomplices to crimes done by their superiors/coworkers. The incentives must be high because, no matter what the rules say about it, whistleblowers will expect to not work in finance again after blowing the whistle. The rules work because those large-scale financial crimes generally are hard to without one’s competent colleagues finding out.
A well designed system, whether a computer or an organization should always have controls that guide things to where they should be. That means carrots and sticks.
In a place like a post office, that means separation of duties, audit, and clear lines of accountability. Things fail when nobody asks awkward questions or looks.
I do think that leadership responsible for a debacle like this must be punished. Their incompetence destroyed lives of innocent people.
The problem here was that as many sub-postmasters were co owned by a husband and wife with children. There was an enormous incentive for one of the sub post masters to plead guilty and say, "my wife had nothing to do with it." Doubly so as the sentences would often be on the cusp of the "suspendible range", a guilty plea combined with contrition is much more likely to get a judge to suspend your sentence if he can.
Of course, this "plea-bargaining" is standard practice in the states but we aren't used to it in the UK.
The issue is that no barrister would allow their client to plead guilty without giving them a little canned speech about not pleading guilty if they didn't do it and having them, "sign their brief".
Holding executives responsible for their misdeeds is the bare minimum in the meantime.
I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work?
I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else.
And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.
That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits.
https://en.wikipedia.org/wiki/Systemantics
now this is depressing. I hate everything and I live in a constant 'everything is shit' state of mind, but this book hates things way more than me.
We’ve had millions of years of evolution to prune out problems, but in many cases the problems were papered over by a kludge that introduced other problems, which were then kludged, etc. No wonder the complex systems we build are such a mess. They’re a reflection of ourselves.
I've used this story a couple of times before on HN, and if it hadn't happened to mother's coworker, I would have thought it a preposterous joke. Mom's colleague was buying something during her lunch break, and had her card rejected. Got the same result in another store, so obviously thought her card had just broken.
Went to her local bank branch to ask what's wrong. The clerk took the card, looked up from her screen and against the evidence in front of eyes declared: "You are dead." Once that kind of flag starts to make its way through our computerised systems, it is very difficult to reverse. It took several weeks to get the error corrected, and in the meanwhile other systems that trusted the bank as a source of truth had picked up the information and started to apply it to their accounts.
Eventually it came to light that a full namesake's estate had had their accounts finally closed, and the clerk doing that had done only a name search when applying the "account terminated, owner deceased" flag to the accounts. Bank never took responsibility for their process failure. They fought for a good time to even have to accept that there had been an error in the first place.
Which incidentally is not entirely unlike the situation in this perversion of justice.
So true. I went through the Wikipedia article, and couldn't escape this feeling:
1) Some errors were made (faulty IT system).
2) Handling of said errors expands enormously into various reports, IT vendor denying their system is faulty, legal proceedings, criminal convictions "tainted" & overturned, etc.
To the point where original errors almost seem irrelevant within the bigger frame of subsequent events.
Imho all the mistakes were made in the processes concerning 2), and 1) is a tiny side-show at most. IT systems can fail, period. What matters is how that's handled.
I transferred 20k from savings to checking. During the process I think what happened is the web banking page froze and allowed me to hold the enter key on confirm that repeated several times. I ended up with 200k in my account that came as a negative 180k balance to my saving account sent to my checking account. (values rounded for story simplicity)
This meant that not only did their online portal not have controls to make sure that internal money transfers are confirmed before processing. They also had no controls to make sure that when an internal transfer is done that the money is there to be transferred.
I waited a couple days expecting a call. Nope. At the end of week I call them. Get transferred around a bunch. The whole time everyone is saying everything is good nothing is wrong, there is no problem. I 100% certain every single one of them was lying through their teeth while they go the FBI on the phone because of the automatic assumption that the computers are right and I am automatically a criminal. There is no way all those people saw a -200k balance in a saving account and thought it was fine. (BTW that means it is standard bank policy to lie to a customer when something is wrong)
So what happened. They immediately zeroed my accounts and froze all my bank activity for three months while they "investigated" Then they only returned 15K of the original 20k that I moved so they effectively stole 5k from me. I spend another 6 months trying to get that money back and did not succeed.
I would have been screwed if I had not been born poor and was familiar with the banking system corruption. I have my income,spending,saving and investing split among different banks and banks accounts in case of something like this ever happened.
The bank initials are WF.
"Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. ..."
My spouse did run into some nonsense some years ago due to her checking and savings accounts being at the same bank and automatically linked and mixed-up. Banks are always offering you super convenient awesome features with "protection" and "connection" and just PLEASE NO - that's how one mistake, that was that bank's in the first place, takes down all your available funds. You'll probably get most back, eventually, but do you really want to live with that for some weeks/months ...
What must not to fail has very rigorous standards, e.g. Nasa, aerospace, and medical devices.
Everything is built like an onion of failsafes.
Beyond that too, I think it's just important for people to realize that most things just don't matter that much. Everything we do eventually has to reach human scale and at human scale, there's so much room for slop and correction that most failures don't matter. Missing a paycheck because the payroll computer dropped your row is awful, but society can be built around allowing people to recover from that. Late payment forgiveness, forbearance, procedures around cutting a new check early.
I think the biggest danger about the "presumption that computers are working correctly" is in it's ability to erode our understand of the fact that "to err is human". It's too easy if you assume the machines are perfect to assume that any errors are therefore malicious, and so it's easy to go from a system where you have "oops my tax software messed up, no big deal I'll file an amended return" to "the only way for a mistake like this to happen is to intentionally put in bad data, so you must be guilty of tax fraud". Or even worse "the computer won't let me do it". I understand you want computers to have check in place that prevent the overriding of procedure and you don't want to give your low level employees the ability to override procedure at will. I also think that EVERY system should be built such that it is possible with a certain combination of people on site at the point of contact to override ANY check in the system to make the state of the system match what it should be. My specific example in this case was a problem with a prescription at the pharmacy. The script had been partially filled because they had stocking problems, but instead of going into the "partial" state, it was closed as filled completely. The tech knew what had happened and knew we still were owed the rest of the script. The pharmacist knew what had happened and agreed we were owed the rest of the script. The store manager agreed. And yet no one could change the state of the system. The only option was to get a new script from the doctor for a new fill entirely. That's the danger of assuming everything is working correctly all the time.
Imagine something with 200 billion individual parts with individual functions (and that’s just the cpu) that is presumed to work perfectly flawlessly billions of times a second for a decade.
By contrast, the crankshaft on your car goes through a bout 1 billion revolutions before it is worn out. Your heart might beat 4 billion times before you die, if you are really lucky.
The precision and reliability we take for granted from modern computing hardware is dumbfounding.
Wait till you uncover what the SW runs on :). From math, processor/memory, rtl design, verification, tools that generate the circuits to the fabs that create the ICs.
And when I submit this comment it appears for others to view.
It is a miracle this entire structure works. It never ceases to amaze me.
Some examples: -- The websites of various airlines (TAP, Vueling, KLM) have often had bugs in them (in the past 2 years when I used them) to the point that they don't allow you to book tickets or log-in with your account. I would say they where broken more often than not. -- Instagram and Google have been deactivating people's accounts (automatically) for no apparent reason. I've had this happen with Instagram 3x in the space of 2 months after which I just gave up and stopped using it. -- Skype, no explanation needed for those who use it. -- Teams fails in many amazing ways no matter how often they "update" it. I've heard stories about the mute button not actually muting, Teams suddenly reversing the order of the letters in your sentence as you're typing in a chat, it just shuts down in the middle of a call, unresponsive, etc etc etc. The amount of productivity these bugs in this software burn worldwide must be significant and the real cost to companies in the form of wasted man-hours is likely higher than whatever they pay in licensing fees to Microsoft. -- Telegram can't handle an external monitor (the scaling gets all messed up causing you to have to restart Telegram to fix it). -- iCloud takes between seconds and an hour or so to sync files even when all devices are on a good connection.
I think they had a button for this on the right click menu.
TBH, Teams is the quintesence of today's SW development state of the art.
Also I can't edit that post anymore to fix it.
And to think that HN is an example of relatively good software that seems un-bloated, super stable, fast and seems to always work :P
Markdown ignores single return because people used it to manually wrap their plain text but HTML can wrap automatically on the fly.
Add two spaces before the return to make a line break `<br>`. Two returns in a row start a new paragraph `<p>`.
But as an engineer I find this natural and efficient! You should prioritize and some bugs are less important than others. If anything more effort should be put in education, health and fundamental research rather than fixing all bugs in all software.
It does remind me though of the book Systemantics: https://www.amazon.com/Systemantics-Systems-Work-Especially-...
One of the core thesis in the book is :
The system itself does not actually do what it says it is doing (The Operational Fallacy).
This people and engineers as well often forget. Just because the system says it does X does not mean it actually does it. People would do well to cultivate a healthy skepticism towards those complex IT systems. They might not fail most of the time but they inevitably will.
So effectively, the Post Office was acting as both complainant and prosecuting authority.
"The Post Office Investigation Branch (IB) is the oldest recognised criminal investigations force in the world. For more than 335 years it has worked to detect offences against the post and prosecute the perpetrators of these crimes."
https://www.postalmuseum.org/blog/the-post-office-investigat...
Obviously, the CPS should have done this in the Post Office case. During the relevant period, the CPS was underresourced and overloaded (it still is now).
Ok I found this: https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...
Numerous people that rabbithole into this case end up running to a nearby bucket. It is... not pretty.
There are thousands of cases that are hidden in the bowels of the post office court system
Oh and the compensation are uh... let say hard to get the money for.
Every transaction the system touched should be inspected, because I'm sure there were plenty of sub-postmasters who took the "shortfall" at face value and paid up without even thinking about questioning the system.
Having agency in a society ( through democracy, markets, shaping narratives in discussions and many other ways) might just require adults who deal with the horrible aspects society produces in a responsible manner. Which includes not designing solutions for a false peace of mind. After all, you yourself (as in every individual) are the one responsible for the consequences you produce.
Personally i hope this might be a matter of outer alignment, as in a problem of not agreeing on shared goals that you can apply to problem solving in a coordinated manner. Which has to include the ability to verify process and outcome (which unfortunately in itself are highly safety critical) and the unwillingness to accept false but pleasant conclusions with high costs for others.
But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.
“The Intel Pentium series of CPUs had two well-known bugs discovered after it was brought to market, the FDIV bug affecting floating point division which resulted in a recall in 1994, and the F00F bug discovered in 1997 which causes the processor to stop operating until rebooted.”
You reminded me of the Things I Won't Work With piece that observed in passing that the behavior of dioxygen difluoride was surprisingly well described by its chemical formula. ( https://www.science.org/content/blog-post/things-i-won-t-wor... )
> But software.... software gets things wrong all the time and the computer hardware just blindly does what the software told it to.
That's not a distinction that's relevant to this article. By "computer" it means a system composed of hardware and software (and which is almost always what "computer" is used to mean):
> The presumption that computers are presumed to be operating correctly, unless there is evidence to the contrary is what lawyers call “a presumption of evidence”.
> This means that a court can be satisfied that a relevant fact can be established just by computer records, unless there is evidence that the computer is not working properly.
> And so when the computer record shows, for instance, a financial shortfall by postmaster or postmistress, the court will accept that as evidence of an actual shortfall - unless the defendant can show that the computer was not operating correctly.
> In short, when the computer record is the essence of a prosecution case: computer says guilty.
too broad a generalization i think.
who told these computers to do what they did?
https://www.bbc.com/future/article/20221011-how-space-weathe....
Also just look at the errata sheet for any microcontroller. Hardware bugs are so prevalent we have a special term and semi standardized document to describe them.
https://apnews.com/article/iphone-15-overheating-apple-softw...
As others in this thread have said, for bits of the system that have clearly defined parameters and get extensive testing (by like, millions of people), I think you can begin with the assumption that those bits are correct: you can assume the CPU is adding numbers correctly, that the compiler compiled the code correctly, that the spreadsheet executed the formula it was given correctly, and it's up to the defense to prove otherwise. Bugs in the CPU / compiler / spreadsheet software would get caught pretty quickly.
But for bits where the right answer is not clear, and / or doesn't get extensive testing -- the formulas inside the spreadsheet, custom-written software -- the assumption should be that it does not work, and it should be up to the prosecution to demonstrate that the software is working correctly.
It does not seem very practical to throw out all digital evidence (eg photos and videos) because there is some chance of misprocessing. Who even records in analog today? What corresponding standard would you demand for analog evidence? Even in analog there were film issues, processing issues, noise, smudges, tampering and uncertainty, people have been dealing with these sorts of confounders for a long time.
We don't generally require "mathematical certainty" for other types of evidence. Personally I think GP was proposing a reasonably balanced commonsense perspective.
I have tried to build models for the schemas that databases have, not their implementation, just the schema, and it is difficult AF. Using z3 for this might as well be trying to start a fire with a pair of rocks.
Yes, in general it is difficult, because then software becomes mathematics. And that's on top of the software doing something useful and what the user wants.
But I think we are now getting the tools to get there. You will still need to become a mathematician, though.
Yes, a schema is a data-model, and a database can tell you whether the row you are trying to insert satisfies the constraints, but it can't tell you whether an instantiation of the database exists, nor can it tell you how to construct it.
There are 2 issues here, first proving that an instantiation exists; and second proving that there's a sequence of insertions that satisfy the constraints. Notice that some tables can have non sat constraints.
This becomes significantly harder when you are dealing with recursive definitions, cycles in the dependencies, foreign keys that refer different tables across the same constrained columns, type compatibility, and on and on
In other words, the space of all databases described by a schema may not even be constructible :upside_down:
Sounds interesting! Although in practice, solving this is probably not that important, because if you cannot come up with some examples of constructing databases for your schema and application, then the schema is the wrong one pretty much by definition.
You can do it incrementally for parts of the model, but that doesn’t guarantee that long range dependencies are satisfiable.
Yes, you then need to show that it is incrementally constructible, which is a different can of worms… the tree view above isn’t actually fully correct because you duplicate entries mapping to the same object. Because of this duplication, the tree may refer to an object that can only be inserted later.
Oh and the encoding must be able to accommodate arbitrary length arrays and arbitrary objects … which makes usage of SMT solvers difficult to say the least.
While I agree that for the average company this isn’t useful, and the average engineer won’t shoot their feet on purpose, what I am doing needs to accommodate all those scenarios because I have no guarantees of sanity.
I have semi-sketched the idea in Dafny instead of Isabelle. Direct encoding of arrays makes it difficult, but I am thinking iterative deepening might just work. I am working on a Z3 approach for this, and I am crossing my fingers.
If Z3 doesn't work, I will try emitting Dafny code instead and hooking up to the compiler.
Have you ever dealt with the public?
The relevant standard ought to have been "beyond a reasonable doubt", but obviously even those who did know better managed to convince the court that doubting the computer was unreasonable. Apparently without manually checking the output against reality, even when the defendent brought their own numbers?
I strongly suspect that the racial discrimination that's being revealed by the press played its part in that -- no need to worry about the computer, or even about reality, if one's mind is set anyway.
You can also not make any statement about reality by arguing with feasibility.
edit: I am unfortunately not qualified to tell you how feasible ways to deal with reality in this case could look. Which doesnt mean that i cant identify some error cases. You also require different levels of competence for designing a rocket and recognizing a non functioning one.
edit1: In regards to the usage of statistics as a basis one should also remember confidence and these cases https://www.science.org/content/article/unlucky-numbers-figh...
Phrased differently, you want them to prove there aren't any bugs. You can't prove a negative so this is a non-starter. There will always be "but what if it requires specific timing" or "maybe it relies on some other software being installed" or "maybe the computer had a virus". It's basically the same reason the government has to prove you're guilty; because you can't prove you're not guilty.
I think the onus has to be on the defense to prove some kind of a bug exists. The interesting question is whether that implies that the software provider should be required to hand over the source code in discovery. It seems unreasonable to force someone to go on a bug-hunt where their future hangs in the balance without access to the source code. Ditto for providing some kind of a test instance; I can see companies being leery of providing binaries to someone trying to publicly prove there are bugs.
However when using a computer application and noticing unexpected and erroneous behaviour, what proportion is the time can the bug be traced back to hardware vs software?
I’m going with about 1 to 1000. I think that’s conservative. Do you see it differently?
Oh you sweet summer child. Hardware has so many little "quirks' or hacks to make it work correctly and it often doesn't! It's often expected that really low-level, "touching the metal" software makes up for these little bumps that happen quite often.
Here is just one example to get you started: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...
https://edc.intel.com/content/www/us/en/secure/design/confid...
Hardware bugs are simply capsuled better so people dont recognize them as such. Its the same mechanism at play as for people who assume software normally functions. They are also just unaware of the inner working.
Stuff is just really complicated and people should manage their expectations. I get that going from assuming to have a solid base to no such thing existing is difficult to handle but there is no sensible way around that. Not doing so gets you stuff like the article.
I am having quite the big problem with this getting downvoted. The poster is spot on here and this needs to be communicated. Not doing so is irresponsible. Overconfidence in this regard is extremely dangerous and there is no nicer way to say this.
You can not reasonably assume that you are able to judge something works reliably without understanding how the error cases look. Especially if these are capsuled from you and or you dont know what to look out for.
This is at the very core of why something like the post scandal was possible. People repeating the same mistake in this very thread is just really bad. The only thing worse would be not telling them this.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
Wow there are some patronising people responding to your comment.
I think this is mostly true for 95% of software and the experience of software developers. But CPUs are full of bugs that only manifest when certain code is run - perhaps the order of instructions or when in certain states. Often these are hidden/fixed in software - compilers/drivers/microcode etc.
By hardware do you mean CPU?
Also, these days, "hardware" is still something that's either got a microcontroller, FPGA, or some other thing that is eother running or was designed with software. The area is quite muddy, especially if you deal with hardware systems of a collection of various computers and hardware components.
And I'm not so sure things aren't CPU or IC bugs that get bubbled up. The amount of Heisenbugs and various other transient bugs make me question that.
In my experience with instrument control systems, the bug could be anywhere from a cable to the UI or anything in between. These systems had to be developed in strict layers to help isolate bugs and also workaround them.
Hardware people are fully aware of their shortcomings, but they also are fully aware that software is even worse.
Far far worse.
It also leads to the same horrible misjudgements as in the article mentioned above. If you dont know what to look for you will come to erroneous convictions by ruling out the error cases you do know.
Related issue, but from the opposite perspective: some years ago I remember the copyright trolls using their own logs of the bittorrent swarm as an evidence in courts. Faking of this data would have been both very easy and pretty much impossible to detect by an outsider. Also they had a massive financial incentive to do so. The fact that this "evidence" was considered in the courts seemed outrageous to me.
A software developer would presume proper working of software if it is being used widely. I would presume JavaSDK or pandas library to be working properly, inspite of the fact that pandas that 3.5K bugs file on GitHub issues. But if I am thinking about integration a less known library or using a less known software I am less confident about its working.
Less ironically, one can expect widely used features of widely used software to have been tested in realistic conditions so whatever bugs there might be are at least known. But it takes an insane amount of testing to cover the phase space even of a moderately complex piece of software: look how large is the test suite of SQLite.
Apparently the only working approach to less buggy software is to compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily. This is the philosophy behind the original Unix. This is the philosophy behind functional programming. This is partly the philosophy behind actor systems (even though reasoning about async interactions is harder). I expect to see more and more of these in practice.
The epitome of that was utf-8 support for a long time: there were a huge number of bugs, but alaphabet based languages would be mostly ok, and most devs would know it was a mess and work around that as much as they can.
To me one should expect any software to be broken in numerous ways, and take the time to check what happens around the area that are critical to them.
> compose software from small, self-contained,isolated, well-understood parts, so that you can reason about the whole more easily.
That looks awfully close to the micro-service approach as well. The approach can be good, but it still gives me pause.
Microservices aside, GP's statement seems to imply composition is like addition wrt. complexity. Unfortunately, in reality it's multiplication. Complexity multiplies with each moving part. There's hardly a better example of that to what we inherited from "original Unix" - chaining small, focused tools in pipes turns into a write-only mess after around the fourth pipe.
Software complexity isn't simply a divide-and-conquer problem, because we pay a penalty for each individual piece, and each piece of glue that connects it with other pieces. That's the whole reason we rely on abstraction and generalization so much - the ability to take a bunch of "small, self-contained, isolated, well-understood parts", put them in a box, and stick a higher-level name to it. Recursively.
But if you have two tightly coupled components which can arbitrarily jump into each other's code, you have (m + n)^2 states.
A four-stage Unix pipe is easy to split and thus analyze and understand, step by step, because each step is reasonably small, and has only one input and one output (presumably). Spaghetti code may be impenetrable, and only work by coincidence.
Microservices are not even about structuring your app; they are about the particular kind of deployment structure for your app. You can make a highly modular app along the lines of DDD and separation of concerns, and deploy it as a monolith for simplicity, or cut it into 2-3 large deployables. (Sadly, there are cases when a monolith remains tightly coupled, but the coupling is implemented as RPC, and the whole thing is purported to be "micropservice architecture".)
So, does the defense get to subpoena the logs, and depose a programmer to explain why some random request returned a 502?
Does the defense get, like, red team access?
Although I'm a lazy programmer, I do think a lot about failure modes. If my freedom is on the line, I can probably think up a few (orders of magnitude) test cases. Happy to provide a shell script or python or golang binary to check. Heck, if it comes down to it I'll provide a stack of fortran cards. Do I get to see the source? That "few" is probably going to move a decimal place or 5.
I get volume is not reasonable in the eyes of the court. Scale is how you(I) find problems, but whatever. Gimmie source, gimme access, I doubt I could beat Knuth, but with my freedom on the line I could give him a run for his money.
How do CVE's count? If I find evidence of log4j do I auto win?
This makes me think of the Jepsen tire fire talk. The system probably mostly right. But there's no way it's correct. What's a "fair" level of access to prove that?
I remember one case where the FBI installed malware via a browser exploit on thousands of computers to get their IPs and one defendant tried to get the source code but the government managed to keep it secret.
Perhaps also relevant would be United States v. Lafon Ellis [1], where the defendant sought access to the source-code of a DNA fuzzy-matching service.
[0] https://www.eff.org/pages/playpen-cases-frequently-asked-que...
[1] https://www.eff.org/deeplinks/2021/02/federal-court-agrees-p...
I'm glad at least in this instance the defendant got access to the source code. He was being helped in pursuing this by EFF so I assume there were volunteers that helped with the analysis. But what if your case is lower profile? What if you can't afford to hire 10 programmers for 3 months to write test cases?
Also I wonder how such a request would go down in a UK court.
It could have been your identical twin doing those things as a silly example.
That doesn't mean the prosecution loses.
The goal is to meet that certainty which is by definition below 100% but high enough that society accepts it.
In theory yes, but you can get some idea of how it worked in practice from the transcript of the trial of Seema Misra [1]. This was one of the few cases where the postmaster pleaded not guilty and expert evidence was presented to the jury. The defence complained throughout the trial that their expert did not have adequate access to the underlying computer records. New documents were identified during the course of the trial, and the defence applied unsuccessfully for the trial to be stayed at the close of the prosecution case.
[1]: https://journals.sas.ac.uk/deeslr/article/view/2217/2151
The post office found that money was "missing". It went after the sub-postmasters. The sub-postmasters didn't know that it was a systemic issue, they were told to plead guilty to reduce their sentence.
Unless you have a credible lawyer, and have a clue that something in the system might be wrong, then you're not going to know.
Anyway, David also posted this to his non-gated "The Law and Policy Blog" at [1].
[1] https://davidallengreen.com/2023/09/computer-says-guilty-an-...
There are more comments there, including this one from me (written for his blog's legal/non-technical audience, not for the HN crowd!):
"There has been a big shift in the practice of computer programming in the last 25 years. In the “old days”, software was delivered on floppy disks or tape, and pretty much had to be correct first time.
Nowadays, everything is on the web, “agile” means the functionality gets delivered incrementally in 2-week sprints. It is regarded as ok to leave out the difficult cases or ship bugs, provided they don’t affect too many users too much of the time. In other words, the software is never really finished.
With that in mind, it becomes simple to rebut the presumption of correctness: if the company’s tech organisation describes itself as “agile”, using “scrum” etc, its software – by design – contains bugs and omissions, both known and unknown.
Fortunately, modern development practices means evidence for this will be in the dev tickets in GitHub, Azure DevOps or whatever other systems their dev teams use."
>To treat the operation of an extensive software system as being a mechanical device is as gross a category error as treating an airliner as being merely a collection of nuts and bolts. In either case we can satisfy ourselves that the elementary parts are performing within specification without having gained any confidence that the complete assemblage will function properly under all conditions to which it will be reasonably exposed; indeed we won’t even be confident that we understand what it was exposed to. Even showing that it has passed a collection of tests isn’t adequate: one needs to see what unfixed issues are being registered against it and how they are being resolved (fixed? Swept under the carpet to keep the numbers below since arbitrary threshold?) The opposing side in case hinging upon accuracy of computer records should have the right to access the issue database, and absence of a well-run system or a large number of issues in it (or evidence that issues have been “scrubbed”) should be taken as a strong argument that the computer records are unreliable.
>Much could be learned by the software industry from the culture of aviation safety, where even minor incidents are followed up by detailed investigations intentionally run in a non-judgemental way and the findings distilled into public reports and recommendations. Similarly, one of the best legal developments in recent years has been the increasing tendency to publish detailed judgements.
https://law.stanford.edu/publications/soundthinkings-black-b...
"I've noticed that responses to attempts to complain about uncorrected technical problems tend to take certain recurring forms ...
What has happened? Someone, in this case a customer though in other cases it has often been an employee, complains that something is badly amiss with some system, receives no redress, and finds that their reports are ignored. So they make noise about it. The response?
1. Misplaced accusations of uncooperativeness. ...
2. Condescending lectures about how `incredibly complicated' the systems in question are ... "
etc. See the post itself for much more:
Also hardware and software are completely codependent nowadays.
Your trust in hardware is hopelessly misplaced.
If I present a digital photo as evidence it would be dismissed by defense lawyers to easily. Then basically any digital evidence would be dismissed the same way.
I can see how it was easy to manage in the 80’s and not anymore in 00’s.
The UK apparently made that explicit. Doubtless others have as well: if we could forego the flag waving and whataboutery, That Would Be Great (tm).
I'd hold up Judge Alsup if I wanted to give an example of a tech saavy legal practitioner.
Assuming it's right, and proving it buggy, is the right principle.
Add to that malicious actors actively trying to work against systems (bank/post office/medical system etc) and it becomes pretty hard to prove that both the software and the data referenced by the software is in fact correct.
Now think about the last time a critical piece of data was recorded on paper with a pen/analogue typewriter and both parties got a physical copy of it.
Last time you made a cash withdrawal did you have to go to a teller at the bank, fill in paperwork, have two copies of that paperwork exist and then that paperwork was digitised or was the entire process digital beginning to end?
Think about how you, if you were an expert witness, would prove that an transaction made on a banking website was not a computer fault.
Do you show a code audit which meets some standards?
Do you have the bank's CISO explain all of the security measures?
INAL but I'm reasonably sure the defendant only has to prove that a system with the same security and audit trail has in fact been faulty and your evidence goes out the door or at least the validity of it decreases.
That and no matter how hard you try there will always be logic errors in programming that will need to be fixed.
Bugs don't have to be root caused and solved in order to call into question the use of some computer situation as circumstantial evidence.
The claimed buggy, externally-visible behavior just has to be reproduced. If there are third-party reports of it, that would help.
That doesn't mean that in general the subsections of it related to any specific court case isn't correct.
Unfortunately disproving something is much much harder than proving something.
Proving that there was a bug in a program is hard but proving that there isn't is nearly impossible and that is likely why the law is the way it is.
This is just another turtle layer because such proof is only vs a "specification", which turns out to also be a program, written in another language.
Maybe another principle would be that a program is assumed to function as proven in the past. The further a given program is shown prior to be correct, the more the burden of proof shifts to the defendant to show otherwise.
The standard does not have to be beyond all doubt. The standard is fine at beyond reasonable doubt. If the prosecution can replicate consistent evidence for and against the defendant using the same computer systems with simulated data, then that is probably enough to clear reasonable doubt.
But the burden of proof should absolutely always be on the party wishing to use "evidence" to take someone else's property, freedom, or life.
Assuming it's right is assuming guilt. That is the wrong principle.
No it isn't. It just means that some situation involving information from computers is "circumstantial evidence".
If programs have to be proved correct, that basically means data from computers could never be introduced as circumstantial evidence.
For instance, guns are normally assumed not to go off unless someone pulls the trigger. If a member of the Crips guns down a Bloods member across the street, he could claim that his gun accidentally went off the moment he was inadvertently pointing a gun in the direction of the enemy gang member. If he could prove that true, then it could be ruled an accidental death. If the defense lawyer does not make that argument, it will be assumed that there was nothing wrong with the gun, and thus someone pulled the trigger.
Well, I looked into this; it's not that clear-cut.