Blame for 'switch from hell' falls heavily on one GM engineer
europe.autonews.com
europe.autonews.com
It is a bit like banking and these "rogue traders" who spend a billion dollars. They are at fault certainly BUT more so it is the banks fault for having such lax measures that they had access to this much money.
No matter how you slice it, this is a management problem, and a cultural problem. It is not an individual responsibility to ensure quality. If it is left to an individual, then things like this happen out of sheer statistical probability. The only thing you can depend on is human fallibility, so your systems should be designed for it; they should expect it, more than anything.
We should all look at quality as a systems, management, and fundamentally cultural pursuit.
Sadly, the individual getting the brunt of all the blame here will only worsen the cultural and management factors that caused it in the first place. Future employees will fear individual retribution, and will make rash, illogical, uninformed decisions in the future out of fear. This, too, will not be their "fault" but the fault of the company, the management, and the systems around them.
Deming. Look him up. There's a reason this does not happen in Japanese car companies. There's a reason that when it does, individual employees are not blamed. The reason is critically important to the success of American manufacturing in the future.
I used to inspect amusement rides for safety when I was in college. Occasionally there would be a defect that would arise that would make me feel it was not as safe as I thought it could be. Managers and owners of course would not want their ride shut down on a busy summer day and would pressure me not to close it but I did not care. I knew if children were hurt or killed I would feel horrible, and be the guy taking the blame or criminal penalty for negligence.
So if you are an engineer for some safety critical system never bow in to management pressure as it will be viewed as 100% your fault when something goes wrong. It is not all this engineer's fault as people have pointed out, but when you put your name on the line approving something it is then your responsibility.
To anyone who read this article and thought what the GP thought, consider that the value of any "thought worker" is inherently tied to the amount of trust you're able to place in them. This is why as an engineer the more you get paid, the more accountability you have.
If you ever work for a place which suffers from institutionalized distrust for its employees you should run. Run very very quickly in the opposite direction.
The process and company structure pretty much made it impossible for the the type of situation outlined in the article to occur.
Just as a 30,000 ft overview of the process... imagine a place where every little change required a minimum of 4 signatures, engineer, QA engineer, Head of QA/Reg, and CEO. This allows not only QA, but also the FDA to determine precisely who was responsible for each change. Imagine further that the signatures have to be on paper, and that giant mass of paper documentation has to be submitted to the FDA, along with mountains of other documentation as well as the software itself, for approval. (But everyone understands that it's also submitted so that the FDA knows who to throw in prison if serious bugs crop up.)
The situation outlined in the article should be STRUCTURALLY impossible as well. At the startup I worked at for instance, build engineering was under QA. This meant that a development engineer could not give the build team a build directive. Only the head of QA/Reg could do that. Also, the head of QA/Reg didn't report to the COO or CFO or Chief Counsel or anything like that, she reported directly to the CEO. As well, she was a frequent contributor at board meetings. There was just no way to pull an end around on her.
That's just a rudimentary outline of the controls we had in place at that startup. Believe me, it actually got a WHOLE lot more restrictive than what I've outlined, but the other stuff is not important to the point I'm trying to make.
And that point is this...
While I realize that for cars the safety standard is probably lower than it is for medical imaging and RTP packages, you would think GM would have those rudimentary controls in place at a minimum ???
Such motivation might come from the fear of losing your job.
But, really, isn't an "engineer", someone pretty LOW in the hierarchy ?
I mean, that's kind of the point, the better the structure and processes... the higher up you will need to go to subvert them. In the case of GM... the processes were so poor that you didn't need to go beyond the engineer level to subvert them. (Assuming the information in the article is to be believed.)
according to the article, this blameless paragon of virtue deliberately hid his actions.
- Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1]
[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
Edit: The bbc report says 15 were fired, half were senior legal and engineering executives. To me the issue stops being DeGiorgio's fuckup, but how so many other folks were incentivized to roll with the fuckup. The fuckup is bad, but the the environment that permitted it and later didn't correct it is the real danger. But don't worry, GM assures us that the Valukas report doesn't find a conspiracy or cover-up.
This is in a way cumbersome and annoying but it is understandable why it is put in place this way.
I used to work in consumer electronics, where I could commit whatever I wanted without any kind of supervision or checks, but this is generally not something you can do when it is a matter of life and death.
Furthermore, when trust is placed in the system and blame not given to the individual, then the momentum is to improve the final quality, not to hide a defect. The problem would have been fixed as soon as it was realized without a need to hide, fear, or cover up. No one would be blamed and no one held responsible, and the problem may never have escalated.
The system was poor, therefore the quality was poor. This was not an individual's fault. Quality is a systems problem, plain and simple.
You can't be reckless in a place where everyone is paying really close attention to safety and regulations. On the other hand if you are in environment where everyone is pretty lenient, it is much easier to slip between the checks.
But, from a journalistic point of view, even if the story is not fabricated, it is a better story to have a villain, than trying to convey to the readership how complex organizations (mis)behave.
It sounds odd, but General Motors is a large and complex entity. Their release system is old and buggy, not to mention never completely overhauled, just constantly having more checks ticked on when ever a recall happens. This is actually true at Ford too, the Big 3 hold on to technology a lot longer then other companies. Ford's time clock system is still ran on emulated System360's using MTS a circa 1967 OS.
Lastly DRE's are more of a projects internal sales arm (they sell the product to management. A DRE's job is to also get the test results from the test engineer, get it to the management, take the managements complaints to the designers/hardware engineers, and relay the tests plans back to the test engineers.
Basically high technical middle management. Their responsibilities at some companies may involve actually writing test plans, sometimes not. I don't believe GM does directly but somebody under the DRE does this.
Ultimately the DRE is just the guy who gets all the forms to all the right people so he can push the plans into the system. And takes all the responsibility when a product fails. On top of that your normally managing 5-6 products concurrently, each taking 18-72 months to clear the system. Your work load is basically endless meetings 500+ email a day, and 100+ phone calls.
1. I use the term 'Chinese' in this phrase to represent any state actor who can rapidly deploy, or incentivise, manufacturing capability at a significant discount to their Western counterparts.
>Enter the likes of Telsa Motors
I don't think we can call Telsa Motors a success yet strictly speaking. When its own CEO states its stock is overvalued we need to reconsider out position on the company. Telsa has only had 1 profitable quarter so far in its entire history.
>I have a hunch there will be others, probably 'cheap Chinese knock-offs
I'm not even gonna talk about problems with Chinese steel, Chinese quality control, Testing, etc. Lets just focus on sourcing. Developing a car's transmission costs roughly 5-10 billion dollars. Its so expensive to do that really no automakers do it themselves, same with engine designs. They share these collectively.
To disrupt the industry you need cash, and a lot of it. Metal dyes are outrageous, 250k to 500k+. The initial investments a company needs to break into the industry is stupid high.
On top of that, as an in-experienced new comer it'll be challenging to gain the public perception of having a higher quality product, as your first attempt may not be.
Products in the Auto-industry typically have decade long life spans, with half a decade spent testing them before release. So your trying get 1 billion in angel funding, for a decade you won't even be profitable?
Fuck! MTS? The Michigan Terminal System? I haven't heard that being mentioned since I was, literally, a kid. Even IBM considers that obsolete, and IIRC even development stopped about 15 years ago!
A lot of the Detroit Metro UUG (Unix Users Group, the LUG is very small) people like to find some of the old tech in the basements of Ford and GM and use it for show and tell :)
But the subtext in the article makes it pretty clear. They are tracking cost of design as accurately as unit costs.
It is largely an "in for a dime in for a dollar" scenario for each internal engineer when they approve a design direction that appears to cut a cost. The larger oversight is unaware of some of the hidden issues and each engineer is motivated to paper over them to some extent to protect their relative standing. The engineers who are completely transparent are completely unemployed.
Its not a culture problem, its a management problem. One person cannot be allowed to essentially design, development, qa, and support. If they are who knows what essential steps are not fully performed if at all? checks and balances fall by the wayside when your both.
One note I remember from some buddies who used to be in similar careers, part costs are calculated not by individual unit prices but the production run. Hence a change you say, that is only a few cents can be hundreds of thousands of dollars.
Resulting in it being possible for the steering wheel to detach! whilst driving.
It's no different than an engineer selection of a single line of code, if it's in the wrong place, crashy crashy.
Here's a much better article: http://www.seattlepi.com/news/us/article/Engineer-s-switch-f...
Perhaps the engineer was distracted by an underground boxing ring.
I don't think that this can be compared directly to the kind of adhoc software development that is routine in many places. Engineering processes have developed over decades to eliminate human error and very, very rarely allow single unjustified decisions made by a single person to propegate through to a finished product.
I'm surprised that in a world where individual car platforms cost billions of dollars to develop there was never a point at which the switch selection was challenged. I haven't read the full report though - perhaps it's not quite as the news article suggests. It would be interesting to hear the point of view of someone in the automotive industry (if any of them are HN readers?)
When the key pops out, the engine stalls, power steering goes out and the airbags do not inflate. Where is the "defense in depth"? A robust system should really require two things to go wrong.
Is it that "key in ignition" is a critical part of the vehicle's state machine?
Not being able to decisively turn off the engine has also killed people, remember the Toyota issue not long ago? We think that in general the drivers panicked and floored the accelerator thinking they were pressing the brakes, but the complexity/difference of turning off the engine with a button---as I recall it had to be held down for something more than a second---vs. turning a key all the way counterclockwise and ultimately pulling it out could have resolved the user errors short of fatal crashes.
Ditto the theory that some or many of these were caused by the accelerator getting stuck on a floor mat. Or an embedded computer and/or components attached to it failing in a bad way.
Compare to the big red buttons on some computers and hopefully in all big machine rooms: there are times, when lives are threatened, that you want an easy way to immediately turn everything off. Even if you add a molly-guard in front: https://en.wikipedia.org/wiki/Big_red_button#Molly-guard
If you're truly standing on the brake, then it ought to work, but you might not think of it. I'll bet a lot of people don't usually think of any gears but park, normal forward and reverse.
The big automakers are ultimately causing way more deaths by not adopting autonomous vehicle technology sooner if you ask me. And they will continue to do so because they realize that autonomous fleets make more sense than every consumer buying a car or two hence autonomous cars mean their business is in jeopardy/going to shrink.
We might also look into deaths associated with air quality/pollution and the number of deaths resulting from it. Its always going to be hard to pin down what number of deaths correlates to how many cars driving some number of miles, but I would put serious money on lower emission vehicles significantly reducing deaths resulting from air pollution. Detroit's reluctance to adopt cleaner technology and its obvious love affair with oil is killing many more than their incompetent switch DREs.
Ultimately, even the switch design is a similar phenomena. While many automakers started to get away from keyed ignition switches altogether, GM has continued to use a super cheap part that's obviously a design from the past. You can't tell me the radio on my grandmothers key ring cost versus their switch is worth 13 lives.
Certainly, don't mean to trivialize those that have been killed by this, just want to point out that Detroit is so slow to change that ultimately its costing us way more lives than just those killed by this switch problem.
1 http://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in...
My main point is that big automakers are doing this same think, cover up and all on a grand scale but because you have to look at it from 10,000 feet to see it, it doesn't get the same type of criticism.
you said you disagree, there were like six points I tried to make, was there a particular set you disagreed with?
Those are important, but not quite as immediate as this matter, is really the issue I had. But you are quite right, those are still important issues.
(shame on those who voted you down, IMO you made a good point but dialogue would be more appreciated than random clicking on a down vote).
Knowing what state the vehicle will be in with a keyed ignition is much easier.
Source: mid-twenties engineer. I can't imagine how grandma must feel trying to fiddle with 'radio on, windows rolling, engine off'.
I can kinda see that, but seems like there are other solutions to that. Say I was driving down the road at 40, and my ignition indicated that the car was to be turned off, you would think some little state machine (or even stateless) would say, oh hey, we are actually moving....maybe don't turn of aux power?
Yeah, I hear your disappointment. But, HN is not (and doubt ever was) a homogenous community. It would be pretty boring and even more of an echo chamber if it were. Roses and Dandelions, one more than the other ;-), all form the HN crowd! Sure, dissing redditization is a part of being a responsible HN citizen, but let's not throw the baby out with the bath water!
(now get off my nice lawn all you pesky whippersnappers) :P