Toyotas Unintended Acceleration and the Big Bowl of “Spaghetti” Code (2013)

As much as I would like to agree with you, people have different value systems. And money talks. So quitting isn't always an option, even though it is the most ethical. Can't pay the bills with ethics.

Handing your concerns to management in writing isn't really a done thing in this culture. I sometimes get laughed at for even suggesting it. And people who quit often take a massive pay cut, are forced into contract work with no safety, or even don't ever get hired in software again - especially in Nagoya where everyone seems to knows everyone.

And when you have a family and a mortgage to pay (selling a house here is also a guaranteed massive loss), it can realistically be a choice between you and your family on the street or doing what you're told.

While the engineers aren't blameless, the system over here is really broken.

?

That doesn't sound applicable to the USA, which currently produces more goods than ever before, and more than any other nation except China.

As hardware limits were relaxed over the years, what you could accomplish in software, even with cheap hardware, grew so enormously that the primary value was increasingly in the software. This was such a rapid transition in the computer industry that it nearly destroyed IBM, it propelled Microsoft to power, and I ended up with a career in software.

Toyota and other Asian "hardware" vendors are in some sense where IBM was. They sell prize-winning hardware and whatever software that comes with it is essentially free. (Apple is a lot like this, which is why their hardware keeps getting better and their software--well, did I mention the hardware is thinner?)

The Toyota "platform" still can't do much in software, but it's growing exponentially. Eventually the software in a car might matter more than the (commodity?) hardware, but that's not the world in which Toyota managers were formed. They are obsessive about hardware, but software is an afterthought. It's for guys who couldn't cut it as "real engineers", as was the case in the American "computer industry" until the 1980s.

I also lived and worked in Japan. I was a strategy consultant, and I was told by some big Japanese and Korean companies (whom everyone has heard of) to emphasize my hardware background and not be seen spending too much time hanging out with the software guys if I valued my reputation.

They occasionally send some of their engineers to our offices (for 6-12 months usually) to learn about our culture. There is a clear disconnect when they send these people. They work hard, no problem with long hours, but they have a very hard time getting into the collaborative nature of our teams, and I don't think it is just the language barrier. I guess I take it for granted usually, but it made me realize how much time we spend discussing things between developers, questioning management, writing furiously on the whiteboard, arguing, trying things out, failing, trying something different, finding problems with other people's code, fixing said problems, etc. It can take a bit of thick skin to absorb such criticism and engage in this process, but I think it is crucial to good software development. Someone who just sits quietly in meetings, then waits for a task to be assigned, and works at that task until it is complete is somewhat missing the process, because sometimes it turns out the task your manager assigned doesn't make any sense, or can't be done with this framework, or that you would have to use 100 global variables to accomplish the task this way.

I am merely speculating but it's possible that a very rigid social and management hierarchy where it is more important to work hard and respect your position in the company than to solve the problem the right way if it involves bucking authority, could negatively impact the ability of developers to innovate and/or produce a quality product.

Ruby.

I'd say one problem is the testing process is not robust enough for software/firmware as it is for mechanicals. The tests revolve around driving the car, and variations on driving, and have evolved over years and decades. Heavy software control is newer, and in my experience was often not fully understood. Chief Engineers tend to come from mechanical. And there are always hundreds of problems to solve and limited time, and the black-box nature of firmware makes it hard to develop tests for.

A development review meeting tends include various mechanical departments, like Drivetrain or Engine or Chassis, many times they bring physical prototypes to the meeting and the problems and proposed solutions are comprehensible. The build testing procedures subject the car/parts to various mechanical tests, and long-lead tests are performed which stress parts till breaking. It's something you develop experience with an can get your head around.

Firmware bugs are not like that -- having just gone through this with my current consumer electronics startup I experienced the black box component first hand. I remember one software based problem we reviewed at Toyota, where something happened like: when the car has been sitting for >3 hours at <50F weather, and the ignition is started when the car is at a 15 degree angle (side of road) and driven on to a flat plane within 15 seconds, the "check engine" light turns on. Even though we had no test for that, we heard about it for customer complaints came through. But the fix is to have the supplier (in this case, but could be the Toyota engineer) fix that problem, with most of the discussion centered on timing and certainty of fix. I don't remember any discussions about global variables or lines of code.

promotion is based on age and length of employment. employment is essentially for life. you can't really be fired and you if you quit you may never again be hired (and i can't imagine whistle blowing would bring you any better). seniority reins supreme and unquestioned. there is a huge focus on hard work and long hours but also on loyalty and face.

contrast this to where and how you're working and think about the differences it would make. my friends were constantly frustrated that, try as they may, they could not elicit feed/push-back from anyone that worked under them. part of this is them being stupid foreigners that can't "read the air" properly, but part of that is people being extremely hesitant to touch anything that could be considered a break in seniority or disloyal. and this was a start-up. these were the f'in cowboys in comparison.

from what i gather, this system worked out pretty good with the long, patient and deliberate development cycles in hardware. when a product cycle takes ten years it might not be that bad to have people running it 'cause they've been there 10-20 years. but it doesn't work very well in software.

In Cobol 85, all variable are global to the program, and statically allocated at program start up. It's not uncommon for a Cobol program to import 10's or 100's of 'copybooks' and have 100-1000's variables in the global scope. They are, however, name spaced after a fashion.

My experience with this came from a similar behemoth mainframe app; 35 MLoc, 15,000 programs, thousands of jobs, running a batch and interactive green screen app that a company hosted. Was the source of about $500M a year in revenue.

Neural nets are well known for being easily fooled [1] ... I wonder if you could create similar situations for self-driving cars.

[1] http://www.evolvingai.org/fooling

Second, DO-178 was originally published in 1992.

We should be adding smarter automatic systems to the cars to assist drivers, rather than e.g. throwing out drive-by-wire linkages and replacing them with mechanics, which are generally less reliable.

There are already cars which will slam on the brakes in any collision which is strong enough to deploy airbags - it stops the car from careening around too much. It should be possible for cars to detect an impending collision and slam the brakes 1/2 second early to prevent or reduce it.

I don't agree with this. Analysis of the source code, especially with global variables reducing the value of analyzing a unit, is not going to imply that it will discover anything about the (presumably millions of) users who exercise the code in live situations daily. Just the inability to CLEAR the source code from blame is a failing in responsibility on behalf of the brand.

The american litigation system has its issues, but I would argue that class action lawsuits are not among them. It allows social change through clear legal decisions when people are otherwise disagreeing on matters much like this.

DO-178B was created in 1989, Ada was created in the 70's, standards for mission and safety critical systems have been around since the 70's, I was reading about Ada and safety critical programming as a kid in the 80's (I was a weird kid).

http://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_s... has a very good overview.

Whether you accept that multiple independent experts in safety critical programming are right or not is up to you however the stuff that came out of the trial painted a very clear picture to me of just how bad this software is.

A cynical person might consider that if Toyota is this poor how many other manufacturers are also as bad.

EDIT: I'm not saying that the UA was caused by this, without the source code and some proof I can't say that however the general setting is pretty terrifying.

If my car went nuts on acceleration the first thing I'd do would be to jump on the clutch pedal and apply brakes (and possibly set the stick to neutral) but that's because most cars here are manual thus that's what we learn to operate†. Pursuing on that line: once at a stop, cut the ignition and disaster is averted. Start again and you're probably fine due to the reboot. Then drive to nearest Toyota to complain, but since "nothing" (i.e no damage, no injury) really happened, you'll get a shrug and a reflash.

† In a country where people learn only on automatic cars, setting back to neutral is most probably not reflex (if mechanically possible at all)