Hacker News Reader

Comment by kens | Hacker News Reader

If I'm reading the testimony correctly, there is actually no evidence that a stack overflow caused unintended acceleration. The idea is that Toyota used 94% of the stack, and they had recursive functions. If (big if) the recursive functions used enough stack to cause an overflow, memory corruption could happen. If (big if) that memory corruption happened in exactly the right way, it could corrupt variables controlling acceleration. And then, maybe, unintended acceleration could occur.

But that's a far cry from the stack overflow actually causing any cases of unintended acceleration.

View on HN

I didn't read the linked article but I did look through the slides, which were very interesting and talked about

- the abhorrent state of the engine control module code,

- the RToS's design,

- the critical data structures right above the stack,

- that those critical data structures weren't mirrored to detect corruption as is standard and as they did for other data,

- that single bit changes in that critical data structure right above a stack can cause the death of tasks in the RToS whose failsafe capabilities were located in those same tasks, and whose death was tested and confirmed to cause unintended acceleration consistent with accounts and descriptions of the event

- that the failsafe monitoring CPU was not designed to detect this failure, and in fact Toyota outsourced its design and didn't even have the source code to it...

Effectively this is the total point of the transcript - that the system was fatally flawed in software design. Any number of bad things could have happened to cause catastrophic faults. Unintended accleration was demonstrated to be a result that could happen on a fault occurance.

> Unintended accleration was demonstrated to be a result that could happen on a fault occurance.

Where was that demonstrated? It was hypothesized that it may be a result, but I'm not seeing anything other than speculation.

It was done in a lab by Barr & his assistants. See the court transcript.

Again, in the linked slides....

The article itself was basic and didn't really talk to the specific Toyota case, but the linked slides did... if you're going to speculate about the trial, look at Barr's slides, not that article..

http://i.imgur.com/IGWUXAS.png

http://i.imgur.com/1ZiVJRC.png

As far as i understood the slides the stack overflow part is only one critical problem.

The v850 has 1...256kBytes of RAM. Let's assume 256kBytes of RAM. Then lets assume they have a Stack of 16kByte. => There are only 983 Bytes of free stack. Then a function call with 5 Int parameters and 10 Int variables needs around 64Bytes. => Only 16 recursive calls are needed to have a stack overflow.

Further because the critical data structure in the os are not secured the os will have no idea that something is wrong.

(In several projects we had to have our critical data 2 times in RAM, once normal and once inverted. In a critical project we even had to do all computations two times, once with the normal data and once with the inverse data. This way you will also find some bit flipps of the ALU, RAM etc.)

edit: In the slides i found that the stacksize is 4kByte => only 245Bytes are free! So you can't make many recursive calls!

i know you don't have any reason to trust me, but I own a Toyota minivan (in Thailand) and I had this unintended acceleration happen to me. I was in stop-and-go traffic, so I only ended up rear-ending a taxi at <5mph but its incredibly scary to be behind the wheel of a car you have no control over. The car's engine was revving+downshifting to overcome my foot slammed on the break.

taking the car to the toyota repair place, they said there was nothing wrong with the car, and nothing strange in the computer's logs.

The problem with these stories is you omit important details - i.e. what was the year make and model of the car, did you have carpets or floor mats in it etc.

And they always always focus on the car manufactuer of the moment. It used to be Audi's, and it's confirmed that a whole bunch of Jeeps actually will do this at car washes due to static discharge (problem's never really been fixed).

I think the most important detail is that he slammed on the brakes and the car continued to accelerate. Cars with a computer controlled accelerator should be able to stop accelerating if the brake pedal is pushed.

I would also expect that he would recognize it if the accelerator pedal had been stuck on a floor mat.

I think it's interesting that even in a small forum such as HN, we still have an anecdote. It's also interesting that car owners still claim to experience the acceleration problem after the two (mechanical) recalls.

It reminds me of the Therac 25 incident, where there were only "anecdotes" to be found for _years_, and the company concluded that a mechanical switch had to be the fault. It's very interesting that we now have evidence that the software _can_ be at fault.

The problem is cars have weird incentive schemes. Namely: fault, because it's related to insurance and they're expensive. If you have an accident, the absolute last thing you ever do is admit fault.

There's also no specific reason to think the driver would identify various mundane fault conditions - i.e. if your car lurches forward and you hit a taxi, then such an impact would be enough to unwedge a stuck carpet.

There's also lots of unknown detail - i.e. if the car was accelerating out of control while you road the brake, and you hit something at 5mph, then did the car stop accelerating at that point? Was the engine damaged and it stopped then? Why not shift into neutral if you're cognizant of the fault condition? Did the car not shift or what happened etc. In these cases make and year model - in Thailand - is pretty important too, since its a market which would have a lot of old used cars. A 1980s model minivan is going to have rather different throttle control.

It's all the hallmarks of urban legend, replete the buy-in line of it being foreign cars - which is how the story always goes and is played with the media. Sure, I'm willing to believe there are real control system issues, but it seems odd to me that no recall notices go out for ECU updates.

Well, if you read the testimony, it seems clear that Toyota is overconfident in their software. They have updated the ECU software, but still as of 2010 they managed to implement the brake failsafe in the same software task that senses the accelerator, so if that task dies the brake failsafe won't work either.

That there have been tens of deaths and millions of car recalls because of unintended acceleration in Toyotas cars is at least not an urban legend. Of course it will be very difficult to locate the exact problem, but this testimony is interesting in that it shows how that can be the case (contrary to what Toyota has claimed) [Edit: And they have actually been able to reproduce unintended acceleration by memory corrupption].

In Toyotas, the accelerator and the breaks are (were, before this) separate systems.

The downside is that hitting the break would not cut the gas.

The upside is that you would need two separate isolated systems to fault in order to have out-of-control acceleration. Even if the accelerator system is broken, slamming on the breaks will generally always win at the hardware level.

> The upside is that you would need two separate isolated systems to fault in order to have out-of-control acceleration. Even if the accelerator system is broken, slamming on the breaks will generally always win at the hardware level.

IIRC, I think for the court case, the brakes for the particular model of car could physically not stop the engine if the engine was going 100% acceleration at highway speed.

> I think the most important detail is that he slammed on the brakes and the car continued to accelerate. Cars with a computer controlled accelerator should be able to stop accelerating if the brake pedal is pushed.

The transcript from the toyota case reveals that a crash on the control thread (managing acceleration, say) would also crash the brake control system until the car was restarted.

In the car at issue in that lawsuit, it didn't have a brake control system in the ECU at all. But yes, in later (2010) models that was the case.

Actually this points to his story being incorrect. If he slammed on the brakes and the car continued to accelerate than he probably slammed on the gas by accident instead. Testing shows that if you slam on the brakes while the engine is pegged at full throttle, you'll still come to a screeching halt. The brake system vastly overpowers the engine even in high powered cars. In a Camry V6 it actually takes just 16 feet further to stop from 70mph with the throttle wide open at 190ft vs. 174ft.

And the brake system is not drive-by-wire like the throttle is, so it's not susceptible to faulty ECU problems.

That's possible, but I don't think it's likely. The feel of an accelerator is very different from a brake pedal which normally will not go all the way down. Of course, more details about the story would be nice.

Btw, there have also been Prius recalls due to faulty ABS software, so while you'll probably always have some control, braking action can certainly be affected by software issues.

ABS is an independent pump. It can fail and you'll still be able to slam on the brakes.

Also hitting the wrong pedal is not exactly uncommon. People don't like to admit they just made a mistake, hence claims of "unintended acceleration". But just like the claims against Audi back in the day, the modern claims against Toyota appear to be unsubstantiated.

I don't think they appear to be, after numerous recalls affecting millions of Toyota cars. That there have been real cases on unintended acceleration related to Toyota is not really in dispute, unless you think Toyota did the recall mainly as a PR effort.

Some people might be unwilling to admit that they stepped on the wrong pedal, but the likelyhood that some random HNer would do that and then make a post about it?

ABS brakes work by relieving _or_ reinforcing brake pressure, so depending on the system you could end up with uneven, little or no braking action after a failure. Or you could end up with locked wheels.

You might want to actually take a look at some of those recalls, they're pretty bogus (like the whole "bring it in so we can adjust the floor mats" thing). And yes, I do believe Toyota did the recall as a PR effort. Take a look at what Audi did instead. They, correctly, stated there was nothing wrong with the cars and they took an insane PR hit and sales plummeted. There continues to be no actual evidence of unintended acceleration in Toyota vehicles.

And I don't think HNers are somehow less likely to hit the wrong pedal or refuse to admit it. There's always that possibility that they don't even know they hit the wrong pedal.

So, there is NO actual evidence? As in, this is just anecdotal, as in the driver, dealer and technician were all wrong in this instance: http://www.leftlanenews.com/toyota-avalon-displays-unintende...

..and the 400% increase of unintended acceleration events (from the testimony) starting in 2004 is also a complete accident, or brought on by some witchhunt?

Aren't you stretching the limits of what is a reasonable assumption in order to maintain that opinion?

you don't have to talk about me in the 3rd person man. I'm right here. you can ask.

If i pushed the accelerator instead of the brake, you can sure bet I would have done more than just "bump" the taxi in front of me.

The brake was fully pressed to the floor. The engine downshifted and revved to try to move forward. Dunno what else to tell you. Sure seemed like the computer going crazy to me when it happened.

indeed. I tried to recreate the scenario afterwards by pushing both the brake and acclerator at the same time (on an empty road of course). The car didn't try downshifting to compensate for my pushing the brake and it stopped the car.

my minivan is a toyota innova. dunno the year, a 2011 or 2012. And this happened BEFORE i heard of any software glitches. And right when it happened my first thought was "MY GOD THE COMPUTERS GONE CRAZY"

edit: oh and one more thing: I'm a 1 foot driver, and i could tell if my foot wasn't on the brake! ;)

my car is a toyota innova minivan. year is either 2011 or 2012, dunno which (no, I didn't buy it for my pleasure, thankyouverymuch).

i mention in another reply: i thought it was the computer's fault, and only months later did I see articles about other toyotas having the same problem due to a software bug.

This is safety-critical software, and the burden of proof rests on the manufacturer. One of the things that distinguishes an engineering culture from a tinkering one is that in the former, it is not acceptable to assume everything is OK just because we haven't noticed anything going wrong yet.

I'm very nervous because in the Toyota case I have previously encountered people posting "proof" that things happened that seemed more to be marketing for their own company.

Isn't 96% of stack usage a bit high? When you are dealing with recursions?

You pay a lot for those cars, can't they at least put better electronic hardware. They probably have less than my phone from 5 years before

> When you are dealing with recursions?

They shouldn't be dealing with recursions. If stack corruption is what caused their failures, inappropriate testing played an important role IMHO.

> You pay a lot for those cars, can't they at least put better electronic hardware.

This isn't how it works for cost-sensitive designs. You don't hear people boasting about how they have a quad-core car computer and how the touchscreens from their motor control are perfect for Facebook interactions.

The way people think about this is, if half your RAM memory never gets used, then you used twice more than you need and your module is more expensive than it should be. CPU use never increases past 20%? It's about 80% more powerful than it needs to be. And so on.

"Better electronic hardware" (in the sense of "more powerful" or "faster") also introduces additional complexity. This means more difficult constraints in testing, longer and more expensive verification processes, additional non-deterministic behaviour and so on.

Not that their system wasn't at fault. It was, but throwing more hardware at it wouldn't have made it better.

Mission critical applications tend to use working, proven, components. Why buy more complexity?

What I find confusing about the article is that it describes how to avoid problems on a completely different architecture. ARM (Von Neumann) vs v850 (Harvard), hard stack exception vs none, etc.

These differences and resulting inapplicable recommendations confound what could be an interesting article.

The article is indeed a mish-mash. Not everything that uses the ARM instruction set and architecture is Von Newumann though. The Cortex M3 and M4 MCU series are Harvard.

Good point, although it was a bit cheeky of me to generalize anything these days as Von Neumann vs Harvard. Thanks for keeping me honestish. :)

Actually I slightly disagree with your "You don't hear people boasting about how they have a quad-core car computer and how the touchscreens from their motor control are perfect for Facebook interactions." statement.

Of course you're right if we're talking engine management/internal stuff, but complaining about the laggy/slow/annoying performance in all things entertainment is quite a well-known first world problem in my circles.

In fact, I'm annoyed by each car I drove over the last 10 years due to their inability to provide sensible hardware (and charge a huge markup for all these 'official' components on top: Think navigation: You get a 3rd party system for a fraction of the cost of the supplier provided one, often providing better features, decent updates, extensibility - while you're stuck with whatever your manufacturer grabbed for pennies).

So in context ("Increase RAM so that the stack doesn't grow into the area where my acceleration value is stored") you're right, this isn't an issue. In general though I haven't seen a car manufacturer that gets consumer electronics/entertainment etc. right.

> This isn't how it works for cost-sensitive designs. You don't hear people boasting about how they have a quad-core car computer and how the touchscreens from their motor control are perfect for Facebook interactions.

I work for a company that makes quad core computers for automotive use and they do end up being used for Facebook interactions among other things like the dashboard etc. The engine management computers will be a separate entity, though. If you look at the big auto shows from the past few years, the car manufacturers clearly do think that this going to be a major differentiator in the next years and it's going to be average consumer models too, not just premium sports cars like today.

But the quad core chips we sell today will be on the road in five or more years. By the time they roll out of the assembly line, the computers will not be spectacular by the standards of that day. A smartphone is 6-18 months from design to production, a car is several times that.

It's not like the car manufacturers were cheap on the hardware.

In a sense they are rather cheap on the hardware.

Instead of using stuff like netbook chipsets they tend to gravitate towards mobile chipsets. The difference between IMX.6 and AMD Jaguar (just examples, you can also look at Intel's chipsets and other boards like Tegra etc) is like night and day. Why isn't the Jaguar used in Mobile phones? Because it can use tens of watts compared to just few watts of IMX.6.

So at least for me it seems the companies wish to save few watts of power usage and few tens of dollars per car.

My car was parked outside in sunny upstate NY a few weeks ago. I started it up, warmed up the engine for 5 m and drove 70mph to work in the cold. The temperature? -5 F. In a few months, it will be 110+ F in the sun and humidity when the car is parked outside.

Things like engine control modules and even entertainment systems in cars operate outside of the environmental ranges that a netbook is designed to survive in. I don't want my car sidelined because of some unreliable computer.

The safety regulations, standards compliance requirements, mechanical, thermal and electrical parameters are significantly more strenuous for automotive ICs than for consumer devices. Does AMD manufacture Jaguar chips that can be used in automotive?

There are also issues of logistics at stake, such as maintenance. Unless AMD is willing to manufacture a certain Jaguar chip for the 5-8 years that automotive manufacturers typically require it, the Jaguars wouldn't even be considered for many systems.

But having some additional capacity available gives them the ability to do field upgrades. The firm I worked for a while ago had to undertake a very expensive hardware refresh because there just wasn't any way to get any additional bug fixes into the field -- they were down to 20 bytes free. In something like a car, that you know people are going to drive for 10+ years, you need that extra space not only to make bug fixes, but also to comply with new legislation (such as brake override), and also to offer a few new features to your customers.

Another valuable engineering lesson; Always Have Headroom. If one designs or operates to the limit there is no margin for error. Resilient systems can get pushed beyond their acceptable limits and recover.

1+.

IMO, You need several sets of limits: standard limits posted to the consumer, engineering limits posted to the techie/maintenance guy/developer/etc, and actual limits.... each of those is comfortably beyond the others. Know the actual limit, but design well under that if at all possible, because the system will be misused.

It would be stupid of me to disagree. One should always leave headroom for bugfixes, future expansion and so on; it's only mindlessly throwing hardware at problems that I disagree with, not futureproof engineering.

Similar situation to why you still get spacecraft using chips from the 90s. They need chips which will remain reliable for a significant lifespan, in hostile environments. Way beyond what you're going to expect for a phone or PC.

Quite a challenge when the consequences for failure could be extreme.

The failures were extreme, both in terms of lives and brand.

> You pay a lot for those cars, can't they at least put better electronic hardware. They probably have less than my phone from 5 years before

Car electronics have a very long development process. When the cars in question (models from ~5 years ago) were designed (~10 years ago), the hardware they chose was probably quite decent for that era.

When the next model of the car is designed, they will most likely end up using the same model of computer (or a successor with conservative upgrades) to avoid having to redesign the hardware and software that much.

The cost of the actual hardware is negligible compared to the cost of the redesign.

What cost? Lives, recalls? This probably is the motivation, but it is a false myopic accounting that makes this calculation. I have personally witnessed way too many embedded engineers cost reduce themselves directly into this situation and do it as a matter of pride. They will use a 2.75$ part instead of a 3.80$ part that forces design trade-offs that cause errors like this. The WRT54 was released in 2002, it had 16MB of ram and 4MB of flash running @ 125Mhz. There is absolutely no reason that the ECU couldn't have been running something similar.

I would argue that Toyota squandered billions with this failure. The ECU needs to be a protocol that can be swapped out at any time to decouple the evolution of the machine.

It is a shame, morally and fiscally that embedded development isn't using safe, provable and verifiable languages.

The WRT54 was released in 2002, it had 16MB of ram and 4MB of flash running @ 125Mhz. There is absolutely no reason that the ECU couldn't have been running something similar.

The WRT54 also runs in a comfy corner of your living room, fails every few years, and crashes.

96% may look high but it's still <= 100%, meaning if their analysis was correct, there's no way it could overflow.

Their software had bugs that could cause a stack overflow and critical data wasn't protected. Sure, it might not be the real cause, but it's a pretty likely candidate. It's not like these errors are unlikely in general. Apart from the occurrence of the stack overflow itself (which is hard to directly observe), the rest of the problem was demonstrated in testing. Memory corruption in the wrong place kills a certain task and that causes uncontrolled acceleration.

Not only that, but the software complexity and amount of globals was so high that thorough analyses or exhaustive testing of the key logic was impossible.

Reply on news.ycombinator.com