Comment by Twirrim | Hacker News Reader

Comment by Twirrim | Hacker News Reader

Parent Full thread

Regardless of whether the acceleration were caused by the software or not, the testimonies for the software experts called in to review Toyota's source code for the case were eye-popping:

http://www.safetyresearch.net/blog/articles/toyota-unintende...

> Skid marks notwithstanding, two of the plaintiffs’ software experts, Phillip Koopman, and Michael Barr, provided fascinating insights into the myriad problems with Toyota’s software development process and its source code – possible bit flips, task deaths that would disable the failsafes, memory corruption, single-point failures, inadequate protections against stack overflow and buffer overflow, single-fault containment regions, thousands of global variables. The list of deficiencies in process and product was lengthy.

>There are a large number of functions that are overly complex. By the standard industry metrics some of them are untestable, meaning that it is so complicated a recipe that there is no way to develop a reliable test suite or test methodology to test all the possible things that can happen in it. Some of them are even so complex that they are what is called unmaintainable, which means that if you go in to fix a bug or to make a change, you're likely to create a new bug in the process. Just because your car has the latest version of the firmware -- that is what we call embedded software -- doesn't mean it is safer necessarily than the older one….And that conclusion is that the failsafes are inadequate. The failsafes that they have contain defects or gaps. But on the whole, the safety architecture is a house of cards. It is possible for a large percentage of the failsafes to be disabled at the same time that the throttle control is lost.

etc. etc.

People quote this repeatedly here, but I'm not sure what it's intended to demonstrate.

Most code is buggy, and the more closely you look, the buggier it is. Most of that same code operates without noticeable error.

That code analysis turns up "inadequate protections against stack overflow and buffer overflow" does not actually suggest that there was any stack overflow or buffer overflow, and people quote this as if it does.

Meanwhile, it is all-but-certain that the overall findings were correct: people hit the wrong pedal regularly, and if enough press attention is given, all of those wrong-pedal-pushers find each other and try to blame the manufacturer instead.

You started with "Regardless of whether the acceleration were caused by the software or not," but I think that's the important thing, and that software is buggy should surprise nobody.

> Meanwhile, it is all-but-certain that the overall findings were correct: people hit the wrong pedal regularly, and if enough press attention is given, all of those wrong-pedal-pushers find each other and try to blame the manufacturer instead.

At risk of beating a dead horse, I do find it interesting how despite the myriad of "driver hit the wrong pedal" news stories out there, only the Prius one starts from the "it had a mind of its own" angle.

I think what most people don't realise is that it's so incredibly easy to be fallible with "automatic" familiar everyday activities (i.e., not maths or remembering facts). I recall in driving school approaching a quiet intersection where I once lost proprioception of my foot and had to call the instructor to use her dual brake because I didn't know where my foot was or could no longer reach the brake pedal (I guess I'd have attempted the hand brake if I were alone).

From this single experience along with running other dark hypotheticals through my head, I am so cautious with many things that it bewilders me that activities such as "carpool karaoke" is not an atypical practice.

But otherwise yep, same thing with software only that the stakes are often lower thanks to the sheer number of non-mission critical projects out there. It's only the mission critical bugs that get the most attention and surprise.

> I once lost proprioception of my foot

That's fascinating. I can say that is something I have never experienced. Does it happen to you with any regularity?

No, I was just using a fancy word I've always liked because it seemed to describe things as best I could.

At one point in my life, I was simply an inexperienced learner driver. It isn't and wasn't a medical condition (for me anyway). I literally couldn't find where the brake pedal was because I couldn't really position my foot in the right place as I had clearly lost track of where it was in the footwell area. I also didn't want to take my eyes off the road and knew I was going slowly enough that nothing bad would happen (because the professional driving instructor had a dual brake, and I had a working handbrake too which I never used - we'll call that risk compensation).

Now all this said, it's incredible how many people are completely unaware of their own inabilities when it comes to mission-critical tasks (the Dunning-Kruger effect). This isn't even always limited to "incomptent" people - experienced professionals also make "simple" blunders. See also Air France 447, or various deceased pro/amateur race car drivers.

Speaking of failures though - another time (also while I was learning), I was reversing out of the car shed with my dad supervising. Before leaving, I complained to my mother (who was just outside the car) that my brake was difficult to press. She replied "just press harder", so I shrugged, obliged and went on with the "supervised" driving practice. Her confidence in me had me convinced that everything was fine so off I went.

Damn, the brake was so difficult to press - I needed both feet as well as some bodyweight on it to get the car to stop at each intersection - it hardly budged each time and felt like a dodgem car brake. I figured at the time that if I just kept things slow and got the handbrake ready, things would be okay? Anyway nothing bad happened; it was only a short session before dinner on quiet streets.

The next morning, my mother went to drive the car out of the shed and was alarmed at how broken the brake was. What was I to know - I was pretty new to driving and thought it was just the car being old and figured it was good enough at the time for my two feet. It was probably bad enough to require towing but I think my dad ended up driving it very slowly/carefully to the mechanic.

That was a bit of a long story, but I'm detailing this to highlight how "stupid" cascading failures are "alarmingly" common - it's not unheard of to exist in both the human side, or the electronic/mechanical side. It doesn't really faze me whether the Prius "acceleration bug" was human or machine induced. Both are often as bad as each other. The only thing we should depend on are multiple layers of redundancy and good systems design.

Edit: When I first started driving, I mulled over whether to go barefoot or not (either is legal here so long as the driver has control). These days, I fortunately can handle either confidently. I'm also much better at sport now (and therefore coordination) than when I was a teenager.

I mean it's a bit depressing but its only eyepopping if you suffer from murray-gelman amnesia about literally every piece of software even written.

Koopman has a talk here on his blog that has more details about how ETCS (Electronic Throttle Control System) is safety critical. "If a driver pumps brakes, loses vacuum power assist...WOT requires an average of 175 lbs of force on brake pedal"

And what was particularly negligent about Toyota's software practices that "more likely than not" caused issues with ETCS.

https://betterembsw.blogspot.com/2014/09/a-case-study-of-toy...

When you're paid to find errors in software, you point out every conceivable bug you can see.

Reply on news.ycombinator.com