Why it's so hard for Toyota to find out what's wrong | Hacker News Reader

Why it's so hard for Toyota to find out what's wrong

washingtonpost.com

I've had a Toyota suddenly accelerate while I was driving it: http://blog.m104.us/2010/03/07/my-toyota-suddenly-accelerate...

Bottom line, this isn't an engineering issue or some big coverup in need of Senate hearings. This is a whole group of interests wanting to put Toyota on the stand for something that, while quite random and frightening, is very mundane and embarrassing. What you're witnessing is a modern day witch hunt where Toyota is the accused and has no way of clearing its name or even addressing the possible problems with integrity. The way we (US media and federal government) are treating Toyota is shameful.

Hacker News readers, especially, should be very worried that the mechanical and computer engineers of Toyota are being told that they must only make systems with 0% chance of failure, even if the driver could be at fault. Entrepreneurs should be scared to death of the kind of liability that the US Senate is laying at the feet of Toyota's management. Zero risk is acceptable, that is the message.

I think a lot of the outrage is justly squared on the possibility that Toyota covered the issue up for years.

I've been driving for a long time, a few years shy of 20, and I even polled my father who's been driving for something like 65 years, at one point as a professional long haul truck driver. If we add up all the times we've accidentally hit the gas when we meant to hit the brakes, we come up with zero cases. That's about 80 years of combined, daily, multi-hour a day driving.

Of course there are cases where this happens, usually with the elderly around a farmer's market. But this doesn't appear to be the case. Other manufacturers have had various dangerous issues, and they take care of them pretty quick.

It's the ones that are denied for a decade, then internal memos are leaked where they coldly calculated the cost of a recall vs. the cost of lawsuits from the loss of human life that stand out in history....anybody remember the Pinto?

It depends a lot on the positioning of the petals. There are some makes and models where it's rather easy to accidentally hit the wrong petal, is it's only an inch or so away from the right petal.

I tend to drive Toyotas, and their petal placement has not been problematic. My mom's Subaru makes it far easier to accidentally hit both petals, which I've managed to do once or twice.

If the placements of the accelerator and brakes are too close , then perhaps that too is an engineering challenge...perhaps more of a user interface one than a "does it work" problem. But an interesting engineering challenge nonetheless.

I've driven a few cars where I did notice the placement was a bit tight, but it just made me extra cautious. However, I could totally see accidentally hitting the wrong pedal if you wore the wrong shoes or panicked or something in those cars. I don't think the Toyota cars under discussion suffer from this design flaw.

> That's about 80 years of combined, daily, multi-hour a day driving.

Not to belittle your point, but you're making a common mistake. Just because "80 years" feels like a big number doesn't make it significant.

You've presented the data point of two (2) drivers. On the other hand, there are hundreds of millions of drivers, just in the USA. Even a vanishingly small percentage of failure cases would still cause hundreds of failures, across so many billions of hours of driving. And easily not appear in any sample of two drivers' experience, no matter how much driving they've done.

You do make an important point. I was simply attempting to counter one anecdote with another. But I think the argument of "the pedals are too close so people are probably just hitting the wrong one", while an interesting thought exercise, digresses from the actual problems:

1) The cars made by Toyota, on occasion, even when under the control of an experienced driver, with years of experience with not only that make/model of vehicle, but even with that exact vehicle, operate in a way which is out of expected operating parameters. This happens with cars from other makers, but #2 and #3 clarify this. http://www.google.com/hostednews/ap/article/ALeqM5gc_pIFqke7...

2) Toyota has known, very clearly about this problem for quite a while and has refused to address it. As far as the public knows, Toyota has done no engineering exercise to discover the nature of the flaw and has so far engaged in finger pointing. When Steve Wozniak found another, similar flaw in the brake system after doing a fairly extensive engineering review of his car, he was initially ignored, then met with surprise at such a wild discovery.

3) Similar problems have been found in other cars, even ones with a strong reputation for quality and reliability. Those makers responded almost immediately and took it seriously. The cat is out of the bag that Toyota was not taking it seriously, or rather that Toyota was calculating public safety vs. the cost of lawsuits. This speaks to a particularly broken internal corporate climate. Which is surprising to anyone who has studied Toyota's corporate management philosophy.

While two data point (mine and my father's) are rightly taken as anecdotal data, there are not, in any real sense that many more reported cases of car crashes -- yet we're all taking it very seriously. When does a collection of events become "statistical data"? My stats prof in uni said that statistics are simply a large collection of anecdotes. The number of people with reported unexpected acceleration problems is actually very small. What's the line that separates anecdotes from "vanishingly small percentage"?

I'd argue that the definition of this line is precisely the mental exercise that Toyota found itself caught up in when it was trying to decide to recall or wait for the law suits. In the end, the company erred on the side of ignoring statistically insignificant anecdotes to their peril.

PJ O'Roorke's "Parliament of Whores" has a chapter about "Sudden Acceleration Incidents" and the investigation and politics thereof that your piece reminds me of (and it's an entertaining read).

http://tinyurl.com/yhkq6k9 - Link to Google books for this chapter (using tinyurl because the URL is truly massive). A few pages missing because it's a "preview".

Excellent link! I've been looking for a term to stand in place of "we're not allowed to blame ordinary people for ordinary problems," and I think O'Rourke's term "blameless citizenry" is exactly that term.

This issue has been going on for 25 years, that was written in like 1986. Are the people with sudden acceleration issues today the same people that had them 25 years ago? Or is it a new generation? I wonder what would be revealed if complaints were mapped geographically and demographically, 25 years later.

I had an automatic 2000 Chevy Cavalier that I drove lightly around my second home when I visited it every few weeks. About 2 years in I came to red light and, in applying the brake, it fought as though I was stomping on the gas pedal. I regularly drive stick so I popped it in neutral as I brought it to the side of the road and the engine kept revving despite neither of my feet being on any pedal.

I turned off the car and re-started it without any problems so I took it to my mechanic down the street and had a friend pick me up. The mechanic gave a full run-through and basically didn't believe me.

I drove the car around for another few months without incident until I was taking a longer drive at night on I-95. The car started unexpectedly accelerating so I removed both feet from the pedals and it kept speeding up.

Long story short, this happened several more times (I have low risk-aversion) despite having the electrical system replaced and having numerous mechanics that I know and trust look at it. Before I just gave up on it (annoyed - I like to know what is wrong) one of my mechanics took it out for a drive just to try it out and had it got away from him as well. We never did figure out what caused it.

Right, I want to be clear: I'm not saying cars don't accelerate on their own. In your case, you had an actual mechanical problem and did the right thing which prevented a tragedy. I knew a Ford Taurus owner with that sort of problem, as well. My point was that the big-story cases with Toyota (and Audi of the past) focus on these harrowing "I was pressing the brake as so hard that I hurt my ankle" kind of stories where we're not allowed to second guess the driver's actions.

There was a case in Minneapolis some years back where a police van accelerated, from a stop, and killed at least one person during some festival of lights thing. What they found, eventually, was that the police department wiring modifications from the stock van (to get the cherries and strobes working correctly) could cause the police vans to accelerate on their own. It wasn't a manufacturer issue, but it was still scary stuff.

That's fair enough. I've driven long enough to see that we all do plenty of stupid-enough things in a fully-functioning car.

Did the car have drive by wire? My pickup ('84 toyota) had a sticky throttle plate. Only just enough to make it idle way too high, not enough for runaway acceleration, (and some wd-40 fixed it), but could happen to other cars too I imagine.

One thing that's really interesting: this is impossible to do in a standard transmission (once you've been driving one for a while). The emergency procedure for a manual is to use the brake while putting the transmission in neutral, which will make the car stop forward acceleration no matter which pedal you had your foot on in the first place.

I've done that. I backed into the door frame of my parent's garage. My dad still doesn't believe I hit the wrong petal.

This looks like it could be a combination of on one hand some people that aren't good at operating a car. Accidentally pressing the wrong pedal. Failing to use the clutch or shift into neutral.

Like this 68 year old woman who said "And I can't remember if I had my foot on the gas pedal … the brake or the gas, I can't remember" http://cbs3.com/local/Montgomery.County.Toyota.2.1454021.htm

And on the other a government that wants to hurt a competitor to their Government Motors.

"It is well-known in our community that there is no scientific, firm way of actually completely verifying and validating software".

Mr Rizzoni, "expert in failure analysis" must have never heard of Ada before. Or, of its use in Airbag steering computers, where the code is mathematically proven to be running correctly.

Honest question: Is there something about Ada that makes it easier to mathematically prove that the code functions as intended?

Ada does have useful things like a formal "denotational" semantics that make it easier than certain other languages.

http://www.google.com/search?client=safari&rls=en&q=...

Years ago, when I looked into it, an "operational" semantics hadn't been defined for Ada, but it seems that the modern language has a defined subset or profile.

"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth

There is a difference between proving a small amount of code that controls airbag (and you just prove the written code, not the compiler, not the os, not the hardware) and millions of lines of code running in a complex machine that run in arbitrary conditions (the car).

In a computer that critical and that simple, there is no OS, and the compiler (or assembler) is tested for the program in question. The hardware design can be demonstrated to correctly execute every permutation of every instruction.

The hardware design can be demonstrated to correctly execute every permutation of every instruction.

Isn't that impossible vis-a-vis the halting problem?

I should rephrase that "every permutation of every instruction executed by the program". I'm not trying to prove that every program is correct, only that my program is correct. That is possible by using a well-defined and fully-proven subset of the available features of the language and processor.

For example, designing your program and CPU as a set of state machines allows you to define all possible states of the system (which are deliberately limited), define all the state transitions, then verify that every input condition for each state results in the correct state transition. Even if you simply brute force your way through every state and every transition instead of using mathematical generalizations, you've still proven that the program is correct.

As it turns out, this doesn't actually work. State derived program analysis has been shown to not be provable for all cases. Particularly when the state-space is very large, and when state transitions are non-atomic in the code, e.g. two or more state transitions in a code block.

The best research I've seen on this was done with Access Control Matrices in the Computer Security field. e.g. can you prove in a general sense that a sequence of atomic state changes to an ACM result in no violations of access control? The answer is, for atomic state changes you can prove that they are internally consistent, but not that they do not introduce a flaw in the ACM.

In other words, because proving software reduces to proving correctness, it only proves that the software is internally consistent. Basically it's a circular proof. It doesn't prove that the software is without flaw.

No, it's just impossible in some cases, and you don't know which cases. If the validator fails to work, the target software is too vague and has to be re-written.

But you can write software that can be proven to be correct. You can also write software that can be proven to stay within certain limits (i.e. the computer will fiddle with the acceleration a bit, but not more than a reasonable pre-defined limit, and not when the brake is on).

No, the halting problem states that it's impossible to know if any given program halts. This says nothing about a particular program. For example, "return false;" halts. But, you can't give me a program that when given another program returns true or false depending on if the passed program halts: that's the halting problem.

Whoa, I didn't that realize Ada was still being used. My dad used it in defensive programming, missiles I think.

http://ertos.nicta.com.au/research/l4.verified/

I realize some cars have a type of 'black box' already, but it seems that with the potential for software problems, a expanded black box that stores vastly greater amounts of data would make sense.

In this case, the computer controls the throttle (no direct mechancial linkage) so it is crazy that the software does not give the brake priority if it believes both are being engaged. A simple 'high braking pressure' sensor that overrides the throttle would be simple enough to be almost immune to any software glitch.

A simple 'high braking pressure' sensor that overrides the throttle would be simple enough to be almost immune to any software glitch.

Fair point. Though pretty much any Engineer (myself included) will tell you that "almost immune" is pretty much a synonym for "bound to happen" :)

I hope that it is a "high braking pressure" rather than "foot glances the brake pedal" sensor, as I was looking forward to one day learning to heel-toe.

It's not that hard. Give it a go somewhere quiet and out of the way, and then just take it easy and practice, I was a few weeks before it was second nature. Just make sure you can actually pull it off in your car before trying it near traffic/children/animals/police.

His point is that you can't heel-toe if brakes kills or misfires the engine (as it often does in karts)

Until you are pulling out into an intersection and the brake sensor misfires, preventing you from accelerating out of danger.

I prefer the physical off switch.

> A simple 'high braking pressure' sensor that overrides the throttle

VWs have had this for a while. I've heard it called the "drive-by-wire fun police."

Super-summary, which is also completely self-evident (IMO), which means the article is superfluous:

Multiple possibilities mean multiple sources, all of which have to be traced for multiple things. Things are also layered pretty deeply.

I still say "sudden acceleration" is different than "pedal was stuck". When the pedal sticks, people say the pedal is sticking, because it's an extremely easy thing to identify (you feel it immediately). Also, intermittent problems are inherently pretty hard to solve.

The article might be superfluous for engineers and programmers, but there are plenty of people who read the Washington Post that don't understand engineering processes. I suspect most people figure that Toyota is just hiding something, and that's why they don't have answers.

And most of those wouldn't be reading articles on it anyway, they're often happy simply thinking the world / governments / corporations / "they" are out to get them. It gives them reason to stay where they are, because they now have proof that it's out of their hands.

It's why smear campaigns work. It's not because the content is accurate, it's because so few people actually look at the content, and instead take what they're told and don't look into it further. Articles like this are mostly meant for the few edge-cases who are actually looking for more information, and they'd find it anyway, and to make the people who know better feel good for educating those poor people who don't.

But this isn't the point you made your initial comment. Sure, most of us here understand the engineering processes of figuring out how and why things break, but that doesn't mean an article in a nationally-read newspaper is superfluous.

WP is behind a paywall? That sucks.

What's Frank Ahrens like as a reporter generally? If this is the quality of his usual stuff I'd be tempted to pay up.

An idea:

Motorcycles and racing cars have had kill switches to fight the problem of stuck pedals etc for ages now. Why not do the same in cars?

It's true, you have the ignition switch, but what about a simple ON/OFF switch right under the driver's thumb? This solves the potential complexity of the key in strange places, the many positions for the key, and the start button. Educate drivers, same as you do bikers and racing drivers.

What about power steering and power brakes? Well, since modern cars are drive by wire, build in an override that forces the engine into idle mode, perhaps with a second override that actually kills the engine (or disengages the transmission) if the engine is still revving high (this could only happen due to a stuck throttle plate, which software can't fix)

If you don't know what to do when your car accelerates outside of your control (i.e., turn the ignition OFF) then you shouldn't be behind the wheel.

There are dozens if not hundreds of reasons a throttle might stick or an engine might suddenly surge and if you can't handle this situation you're a threat to everyone within striking distance of your vehicle.

Whether Toyota is to blame for these cases or not, I think anyone who testifies to losing control of their vehicle for these reasons should have their license revoked on the spot.

Turning off the ignition will kill your power brakes and power steering, and may lock the steering column. Not good if you're in traffic or generally not on a straight, flat road.

Shifting into neutral while standing on the brake pedal (and clutch if you have one) seems to be the consensus on what's safest.

I've never driven a car that locks the steering wheel when the ignition is turned off (you have to turn it to "lock").

Shifting into neutral is good, except that a racing engine presents dangers of it's own. Killing the ignition removes this risk as well.

I'll go on to say that if you can't steer or stop your car without power assist you have yet another strike against you as a operator of an automobile.

When you're in an unintended acceleration situation, you don't want the only solution to be one that introduces several other sudden, unintended changes in the behavior of the car.

And in a panic situation, you're not going to be able to reliably turn the key only one click counterclockwise to the off position instead of two clicks to the lock position.

And in a panic situation, you're not going to be able to reliably turn the key only one click counterclockwise to the off position instead of two clicks to the lock position.

As I stated originally, if you are incapable of handling a vehicle in an emergency situation then you should be relieved of the responsibility.

For the record I have performed this "complex" operation under these conditions (and on a motorcycle as well) and I've met more than one other person who has done the same.

a racing engine is probably not as dangerous as racing down the road at 120mph. Engines with rev limiters (all modern passenger car engines) will not exceed redline, and any well maintained and designed engine should be able to redline for a minute or two without locking up, long enough for most emergencies to come to a close (regardless the outcome)

This is not to say killing ignition is a bad idea, but it does not seem to be required. In any case, the worst things that can happen to the driver that I can think of is fire or accidental re-engagement of the tranny. Only real way I'm aware of for an engine to blow and hurt somebody is when a motorcycle shoots a piston straight through the head of the engine and up into the rider. I don't think transverse V6's can launch pistons through the head AND firewall.

Also while the steering wheel may not lock, power steering will vanish. I want to say the driver will just have to deal, but an 80 year old probably can't turn the non-power assisted wheel of their 4000lb luxury car, and that's not their fault persay. Which brings up other questions. This is a dilemma, and partly why I have never truly liked power steering.

Reply on news.ycombinator.com