Hacker News Reader

Comment by taylodl | Hacker News Reader

"This was a couple of software engineers who put this in for whatever reason," - I'm not buying it. What would be their motive compared say to VW's motive? VW has a heck of a lot more motive for falsifying emissions than do "a couple of software engineers." Even if you were to argue the engineers were somehow fearful for their jobs that's still VW's problem because they're the ones who created that work environment instilling that fear.

I really expected VW to do better in handling this. Some business analysts are saying VW may be out of business within five years time. Based off this testimony I say good riddance.

View on HN

No way could it be just a couple of software engineers. The engine is a system, and its the engine that has to pass the emissions tests. So that means the whole engine development team had to be in on it, including their technical management. They in turn would not have done such a thing if they had not believed that senior management wanted it, and it is the responsibility of senior management to make it clear what they want.

When this CEO of a company with 600k employees says "it's some software engineers", I assume he might actually be honestly referring to what we would call engineering managers, perhaps with hundreds of indirect reports, 2-3 levels down from the CEO.

I find this plausible...

(Particularly since these hearings seem designed to score political points rather than to find out what actually happened.)

The (now resigned) CEO prided himself on being an engineer and in depth knowledge. I don't believe he didn't know how VW was able to market clean diesel engines in the US without having a clean diesel emission system for them.

His big thing was to be the largest car maker and he needed to juice US sales to get there. He had to have spent a lot of time on US strategy. The shitty part is that he succeeded--they became #1 (this year!) and he has a huge pension to retire on.

http://www.volkswagenag.com/content/vwcorp/content/en/the_gr...

"Winterkorn’s professional career began in 1977 as a specialist assistant in the research division "Process Engineering" at Robert Bosch GmbH. From 1978 to 1981 he headed the refrigerant compressor development group "Substances and Processes" at Robert Bosch GmbH and Bosch-Siemens-Hausgeräte GmbH.

In 1981 Winterkorn joined AUDI AG as assistant to the Member of the Board for Quality Assurance. Two years later, he assumed responsibility for "Measuring Technology/Sampling and Test Laboratory" at Audi. At the beginning of 1988, he was made departmental head of "Central Quality Assurance", and in 1990 Head of Audi Quality Assurance.

In 1993 Winterkorn became Head of "Group Quality Assurance" at Volkswagen AG and was appointed General Representative of Volkswagen AG in March 1994."

These would be his formative years. It seems he focused on processes and quality rather than actual technology, IMO.

My impression is that manufacturing technology is still technology, and manufacturing engineers are still engineers.

Absolutely, you don't precisely robotically assemble millions of vehicles without a little engineering and technology getting involved.

So much for quality assurance!

He was also known as a micromanger, I just don't see how he didn't somehow figure out that his entire clean diesel strategy was bullshit.

QA, particularly in Germany, is all about processes. Not deep technical understanding. It bores the hell out of any competent engineer.

fact is the whole report chain is rotten. who wrote the code, who told him so, and indirectly most engineers that had a reasonable knowledge to be in a position of suspecting a foul play.

you can't get this running for 6 years without widespread cooperation, or at least willfully turning a blind eye.

those engines ran loads of miles on benches before being put into cars, and it's a large group running multiple cross tests on many parameters including fuel consumption and efficiency and operating temperature, which should give anyone within the sector a good rough idea of how an engine performs emission wise.

However plausible it may be, it is still not excusable.

The company should have processes in place that can verify the claims it makes. Plausible deniability is really just a claim of neglect.

Edit: I'm being rate-limited, so I'll respond to the points below right here:

>What processes should be in place, exactly, to do what?

How about driving a vehicle in the manner most of their customer's do and examining the emissions?

>Quarterly polygraph tests performed by all engineering managers

How would that help exactly? Polygraph is demonstrably unreliable.

>to assure top management they haven't broken any laws?

It's mostly irrelevant how assured top management is about anything. What matters, in this case, is how far out of spec the vehicles Volkswagon produced are. That is what needed to be verified: specs vs. facts.

What do you mean?

Let's assume the individial he is talking about is a manager of a group of hundreds of engineers working in this area. This means this guy's direct manager most likely would not be in a position to inspect the work of people 2-3 levels down for any irregularities.

What processes should be in place, exactly, to do what? Quarterly polygraph tests performed by all engineering managers to assure top management they haven't broken any laws?

For something like this an open door policy where someone can send issues up the chain would have work over the 6+ years this policy was in place.

> What do you mean?

All you need are strong whistleblowing protections and mandatory independent investigation of any claims made.

Do you really think of the dozens of people who had to be aware of it none of them would have thought "this is really dodgy" and wanted to push it up the chain?

At the end of the day, that manager is still responsible for everything that happens under them. That's the justification for giving them the big paychecks, isn't it?

I thought the justification for their big paychecks was the value they brought.

He's the CEO of the American branch not VAG

My mistake. Still, my argument persists.

> I really expected VW to do better in handling this.

Ignore whatever comes out of the VW USA guy. The investigations in Germany will most likely be more fruitful especially because police is involved.

Exactly. The most devastating part of all of this for me (a US citizen) is that the German CEO resigns in disgrace and there are ongoing criminal investigations in Germany, but in the USA the CEO holds fast and blames engineers.

It seems like the German government is more competent that the US at enforcing laws against corporations, even when those laws are USA laws...

*EDIT: This comment is poorly worded. What I mean to say is, Germany is doing a better job at holding PEOPLE AT THE TOP responsible, whereas the USA is predictably allowing CxOs to plead ignorance and hide behind the corporate veil. Meanwhile in Germany, http://www.wsj.com/articles/german-prosecutors-open-investig...

You realize this is the exact opposite of what the rest of the world is saying, right? Is this the first article you've read on the matter? I've been following it with great interest, and many others have expressed shame that it's the US that is catching a European company cheating, and threatening prosecution.

This all happened because of the work of the West Virginia lab, plus CARB and EPA's investigation, and the EPA's refusal to issue an emissions certification to 2016 diesel VW passenger cars which has led to likely tens of thousands of un-sellable cars sitting on US dealer lots.

The German government has been shown to have looked the other way repeatedly and to be complicit in fighting on behalf of the automakers instead of the environment.

The European emissions and mileage tests have been shown to be an utter and complete sham.

The story you're peddling is really the exact opposite of reality. Take a look around at the other stories that have been posted on HN on the matter.

actually the european started this asking for help since they had the equipement for static test, but not the equipement to fool the defeating device into thinking it was actually on a road.

Good call. It wasn't European governments, and it wasn't an attempt to punish automakers for malfeasance, though.

The ICCT (an NGO) that appears to be based on Europe, commissioned the West Virginia study to try to figure out how more stringent US Diesel standards were being met, so they could encourage automakers to meet the same standards in Europe.

http://www.detroitnews.com/story/business/autos/foreign/2015...

"In 2013, his organization [the ICCT] commissioned a study of VW diesels by West Virginia University after questions were raised about European diesel emissions standards and whether European vehicles were emitting too much nitrogen oxides linked to smog. Testers looked at three diesel cars for the U.S. market: a 2012 VW Jetta, a 2013 VW Passat and a BMW X5. The group expected the cars they tested would perform better than those in Europe, because U.S. regulations are tougher. They were surprised the two Volkswagens had significantly higher-than-expected emissions, while the BMW performed well. They were so certain they had done something wrong that they tested the cars two more times with similar results."

This was an engineering problem. There may be an assumption that while there are about a dozen people in the US who work for VW in an engineering capacity, the US part of the group is in essence purely a marketing operation.

Also, it implies a couple of engineers could slip through arbitrary code to be executed under selective conditions. That's terrifying.

But also probably true, regardless of this specific situation.

From what I know of the car industry, coding standards are very low. I don't think any car companies would look good if their code was audited by professional software engineers in other fields.

http://www.ldra.com/en/software-quality-test-tools/group/by-...

The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C https://en.wikipedia.org/wiki/MISRA_C

Today every thing you do with your car from using breaks to deploying air-bags is done through the computer, there's no way in hell that the motor industry would be releasing poor code "intentionally" because it would cost them billions in liabilities.

And you're not familiar with what happened when a 3rd party (Michael Barr) was allowed to FINALLY look inside Toyota's engine code and determine the cause of the "Unintended Acceleration" problem:

"Barr checked the [Toyota] source code against MISRA’s 2004 edition and found 81,514 violations."

http://www.safetyresearch.net/blog/articles/toyota-unintende...

The model year 2005 Camry in question from the trial would have been made available in 2004, and the electronic throttle control system was designed prior to 2004 (IIRC, the ETC design was carried forward from the previous model). How does it makes sense to apply MISRA-C:2004 in this case, especially when the Toyota ETC in question was admittedly not designed under MISRA-C:2004 nor prior MISRA-C revisions?

It is inaccurate to claim Michael Barr determined the cause of Toyota's unintended acceleration: he proposed a possible failure mode that was persuasive to a lay jury. His proposed failure mode didn't leave a DTC, so there's no way to actually know if his proposed failure mode actually happened in this case (or in any other).

Barr always struck me as something of a charlatan trying to bamboozle a jury.

I remember when this was posted on HN before, what I always wondered about is the discrepancy between the violations that NASA found and Barr.

NASA found 7000 and change Barr found over 81,000, I'm not sure whats the size of the code that was reviewed but that's allot, even if you count the fact that half of them might be silly like naming convention violations that still leaves quite an odd number for actual violations especially when considering that the code for an embedded system can't be that huge.

And the 10,000 global variables thing, well 10,000 sounds like a huge number but there wasn't any mention if they were a) necessary, b) implemented correctly, and c) out of how many variables in total? if the code they've tested has say 1 million variables then well.... And how they described global variables is also weird, they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks, but that might be true only if everything is running within a single application.

If the breaks run as an independent application well then only every function within the break software (depending on the language and how global variables are implemented) could potentially access those variables.

To me pretty much that entire explanation seem to be constructed to sound worse than what it is, which is why I think that expert witnesses should only be allowed to be appointed and called by the court it self (as in by the judge so they and the jury could understand the technical details better) and not by either the defense or the prosecution.

> there wasn't any mention if they were a) necessary,

They were not. It is never necessary to have 10,000 global variables. If you have 10,000 global variables you are systematically Doing It Wrong. (That's not to say it's uncommon. From my experience I strongly suspect that it's a very common practice.) As one of the witnesses put it, quite correctly IMO:

"And in practice, five, ten, okay, fine. 10,000, no, we're done. It is not safe, and I don't need to see all 10,000 global variables to know that that is a problem,” Koopman testified.

Trying to justify this is like trying to justify not having a bug tracker. Which, BTW, they didn't.

> b) implemented correctly,

There's no way to check 10,000 global variables to see if they're implemented correctly, and that's the entire point.

> and c) out of how many variables in total?

That is completely irrelevant.

> they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks,

This was the ECU, it only controls the engine, not the radio or the wipers. It has tasks for stuff like monitoring the engine and wheel speeds, the accelerator and brake pedal positions and controlling the fuel injection. And all of those tasks were only an extra header file and a typo away from stomping on one of the 10,000 global variables belonging to another task, but that isn't even the issue. The issue is that there is no way to trace the flow of data in the system because it's completely unstructured.

> but that might be true only if everything is running within a single application.

Welcome to the world of embedded RTOSs, where everything is running within a single application. It's not a PC. "Applications" aren't a thing. There's certainly no memory protection.

    > Welcome to the world of embedded RTOSs, where
    > everything is running within a single application.
    > There's certainly no memory protection.

Generally, I agree, even though there have been quite a few chips already that have a primitive "Memory Protection Unit" that typically only distinguishes two processor states and you'd be able to protect the working data of e.g. the scheduler and your watchdog from the rest of the code running on your system. Which is much better than nothing!

But in the end the important consequence is: Having everything in one address space encourages people to violate the concepts that memory protection between tasks on your PC typically enforces, e.g. tasks reading from, and writing to other tasks' data structures without proper synchronization; instead of using proper IPC mechanisms like queues. Which is probably what these 10'000 global variables were used for.

And this would be why Toyota engineers _forgot which direction their stack was growing in_ and shit all over MISRA-C (see https://news.ycombinator.com/item?id=10355200)?

I've worked at a couple different companies that made safety critical software, and I have to agree with joezedeco.

Dav3xor's Law -- Code Quality is Inversely Proportional to Risk.

And 2 of my friends worked for BetterPlace (RIP) and it took years for Nissan to accept the code which had to be rewritten to comply and certified by an external 3rd party. And that code wasn't even "mission critical" in terms of road safety.

Anecdotal evidence is anecdotal.

Any theories why? Maybe because these big systems are outsourced to multiple low-ball vendors and their stitched together?

I have seen both very good code and very bad code. Standards are in place for making it good but it really depends the most on who's implementing it and the SW team culture.

The code might even have been outsourced. I'm not trying to disparage outsourced developers, I know what I might do if the user story was something like "Needs to pass emissions when attached to test system."

"As an emissions test, I want to pass the car."

In that case it would probably be "The system shall pass all emissions tests"

Under all conditions? Under normal conditions? Low-temperature? High-temperature? At 10,000 feet? Only when attached to the test harness?

Actually the usual form for this kind of requirement would be "The system shall comply with <emission regulation>".

It shouldn't matter. Another test process should have caught incorrect code. The car should be tested to the specs and the specs should reflect the regulations.

If that's the case, the execs are even more at fault for not having proper controls in place to prevent rogue employees from jeopardizing the entire company.

Welcome to real life.

And, supposing this is true, would you actually be surprised by this? This is a car company, not a CI and code-review obsessed software company in Silicon Valley.

> This is a car company, not a CI and code-review obsessed software company in Silicon Valley.

I read that as:

> This is a company whose products are high-velocity 1500kg chunks of steel zooming around in public, not a company that sells an intangible virtual good/service

---

Yes, I know software is historically poor with car companies, but we should still expect better. Let's not just lower our standards because of cynicism :-)

I found products made by car companies to be in general more reliable than those made by software companies (either in Silicon Valley or not). I trust my life to my car (software included) every day. I wouldn't do that with any software of my smartphone, not even the OS.

Doesn't it blow your mind that there are no "standards of engineering" or whatever for software? There's no licensure body for software engineers who build software running your car, and therefore no accountability on a personal level.

When an engineer builds a bridge, she has to personally sign off on the bridge, saying it's safe, and is risking not only her professional career, but I think she can also be jailed and held criminally liable if the bridge kills people due to negligence.

It blows my mind, at least, that no such thing exists for software.

Does any one really thinks that true?

http://www.ldra.com/en/software-quality-test-tools/group/by-...

https://en.wikipedia.org/wiki/Motor_Industry_Software_Reliab...

https://en.wikipedia.org/wiki/MISRA_C

https://en.wikipedia.org/wiki/ISO_26262

None of those are what I described, though.

> Doesn't it blow your mind that there are no "standards of engineering" or whatever for software?

ISO, IEC, etc.

> There's no licensure body for software engineers who build software running your car, and therefore no accountability on a personal level.

MISRA, SCSC, etc.

Combine my individual sentences, as I did, and try again.

There are many standards of engineering involving software.

- https://en.wikipedia.org/wiki/ISO_26262

- https://en.wikipedia.org/wiki/IEC_62304

- https://en.wikipedia.org/wiki/MISRA_C

The big problem with all those standards, frankly, is that you've gotta pay money to actually evaluate them.

There aren't any guarantees that they'll be useful, that they'll match the modern development processes in your language, that they'll fit your problem domain, etc.

Those standards are there primarily to make the publisher a buck--not to represent the codified wisdom of up-to-date practitioners in a field.

Until we've got a truly open-source standard for people to code against, we should stop wringing our hands about these things.

Not sure whether software is the domain that standards should be applied to. Software is just a tool.

If you use software to build a bridge, bridge standards should apply. If you use software to build a car, car standards should apply.

If you use software to build a fart app, fart app standards should apply. (Which frankly, don't have to be very high.)

If you could have perfectly safe software, without requiring individual engineers to be licensed, would that be acceptable to you?

I'm not asking if you think such a thing would be possible or not - I'm asking if you would accept an alternate means of getting what I think we both want.

A company I worked for had a permit to practice and required engineering sign-off for each release.

And a German one and from experience dealing with German companies software isn't regarded I the same light as what they would consider "proper engineering"

Someone else posted this higher up:

Then you don't know the motor industry very much as it has very strict coding standards like ISO 26262, the code has to be audited internally and by certified 3rd parties with every release to meet the development cycle requirements of the standards. http://www.ldra.com/en/software-quality-test-tools/group/by-.... The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C https://en.wikipedia.org/wiki/MISRA_C

But see the reply: https://news.ycombinator.com/item?id=10355200

I wonder what the NTSB thinks about this. This is a clear safety issue.

US vehicle safety is governed by NHTSA, and they are already using this to push for further regulatory power.

http://www.detroitnews.com/story/business/autos/foreign/2015...

Remember the Toyota code, or at least the review, that was leaked a while back? I don't find it surprising at all that it would be possible to sneak through some stuff that no one knows what it does. And in that kind of environment, people generally don't mess with stuff they don't know what it does, cause they don't know what it'll break.

Just because VW might have more motive to do this doesn't automatically mean that they're lying when they say it was the fault of individual engineers. Guilt is not determined by "who has the most motive", it's determined by who actually did it.

Now it may very well be true that VW is lying and it was a corporate decision. But it's not completely implausible that this really was the decision of a few individual employees.

Other commenters here are expressing surprise that engineers could "slip through arbitrary code" to do this. But isn't that kind of the job of software engineers, to write code to do things? I would hope they have code review going on at VW, but code review doesn't always catch everything, or maybe several engineers who were working together would review each other's code.

"But it's not completely implausible that this really was the decision of a few individual employees."

Oh, I fully believe it was the decision of a few individual employees. I just don't believe those individuals were the actual coders. Coders will code to the spec they are given; problems are far more likely to lie in the spec layer.

It is possible it will turn out the coders should have known better. But it isn't even that hard for me to create plausible circumstances in which the coders didn't have any reason to believe they were doing anything wrong. Remember, we're only seeing the end-results, but it's not hard to imagine a combination of specifications where the programmers are assured each little step is legal and OK, and a combination of other optimizations combining together to produce this result, with no one person quite understanding the interactions.

And just to be clear, I work a lot in security, on the defense side. I'm familiar with taking the possibility that coders will be actively hostile seriously. But I'm also very familiar with seeing five intelligent people each acting according to their own best info and with their own best interests at heart interacting to produce blithering stupidity. It's not always as simple as it looks when you're just looking at the outcome.

The article did say

> The German automaker has suspended 10 senior managers, including three top engineers, as part of its internal investigation.

So it could be a few engineers and their managers that colluded together on this.

Like NASA, auto companies like VW produce a TON of documentation on all phases of design and engineering. This is especially true when it comes to regulation-sensitive processes like emissions testing.

As someone who has worked in a comparably highly regulated industry (pharma) I can state with full confidence that Mr Horn is a lying bastard.

> Like NASA, auto companies like VW produce a TON of documentation on all phases of design and engineering. This is especially true when it comes to regulation-sensitive processes like emissions testing.

Bingo! A lot of HN is acting like "it could slip through the processes we have at work so I can see how it could happen here" without realising that the processes, documentation, and independent review requirements are 100x more involved and thorough for critical car firmware than they are for a basic software shop.

Yes and no. Remember the Toyota firmware fracas and the bizarre coding practices (100s of global variables anyone?) that were eventually revealed. Adding heavyweight process doesn't necessarily solve more problems than it creates.

I'm not sure emissions controls are considered "critical firmware" in the same sense as systems that could actually kill the occupants of the vehicle if they fail. Or rather, I hope they aren't.

If there's three or more engineers, said engineers could have colluded to put the code in.

They will blame some small man to avoid prosecution of higher staff. I bet this was done off the records - so the Software Engineer manager have no proof this was ordered or suggested by upper management. At worst they will get 1-2 engineers arrested while they will move quietly to other companies that will be willing to use their experience and connections.

Of course. Fall guy 101. Not some tinfoilness, but expedient politics of surviving corporate death pits.

The sort of shops where most every discussion is double top-secret: happens in-person, quietly, behind closed doors and the email trail is kept to strictly vague coordination details, at most, since it's subject to discovery. Oh and information is hoarded, because details create weaknesses.

Arrested? Remember that cars don't pollute, drivers do or something like that. At the end of the day, what the engineers in the trenches did was design a device that is illegal to sell in some other country. It is correct that the blame goes to a place higher up, where the decision to sell was made. Which is what is actually happening at Volkswagen proper. The surprising part is that the VW USA CEO seems to be completely out of the loop about this.

Exactly. I don't buy this one bit. If those engineers feared for their jobs because the cars wouldn't pass, I bet those engineers could have left VW and found better jobs. Nobody needs that kind of stress.

I think it is cowardice to blame them.

This is probably a little reactionary, but can we just stop for a second and admire the difference in American CEO's versus his European counterpart? The latter resigns immediately as a way of accepting he is ultimately responsible, the former points fingers. This is a blatant example of why the tide of opinion is turning against those with the most wealth.

Agreed. However I do feel that the European CEO got off easy by simply being able to walk away.

AFAIK the German government is still investigating the issue. He may have walked away, but that's not necessarily the end of the story.

It's pretty close to the end. Fines will be levied against the Volkswagen company, not to him personally. His stock options will become worthless but he's still set for life. I think in this area, at least, China does it right. Shoot the guy as a warning to the others.

The Staatsanwaltschaft Braunschweig is leading the criminal investigation and they said specifically that they are not after Martin Winterkorn, the press reports.

That chapter may not be over.

The cars were designed in Germany, weren't they? (If not, excuse my shock to find we've been subjected to misleading advertising.) I doubt the engineers reported to the USA guy.

Well, not really.

The cars sold in the US have nothing in common with the European models but the name and engine.

The US models have different sizes, different engine selections, are fully assembled in the US, etc.

Some example images:

Interior: http://www.myturbodiesel.com/images/b7/tdi/2012-vw-passat-eu...

Exterior: http://www.myturbodiesel.com/images/b7/tdi/euro-passat-diffe...

Seats: http://www.myturbodiesel.com/images/b7/tdi/interior-differen...

And in this 2011 discussion there are already discussions about how VW is able to handle NOx without AdBlue in the US variation of the car: http://www.myturbodiesel.com/threads/differences-between-201...

Wow, that's amazing. I would never have thought it was sensible to make all those tiny changes from one market to another. I guess it's kind of like the changes they make between model years...

>If those engineers feared for their jobs because the cars wouldn't pass, I bet those engineers could have left VW and found better jobs

But if they could just go out and get jobs, why would they fear for their jobs in the first place?

Right. And those tests and regulations exist in the interest of public health, not product benchmarking. So the issue is bigger than lying about some performance metric to compare more favorable to competitors. And it's also not a domain that's auxiliary—this is engine technology, the very heart of the company and the product. And good god, they built an entire brand campaign around it.

I really do feel this is analogous to some large pharmaceutical finally admitting that they cheated clinical testing on an important drug. You can't just pin that back to a couple of "rouge scientists".

VW will not be out of business in 5 year. Nobody will remember this in 5 years. All the major automakers have manufacturing scandals in their past that resulted in actual deaths, not just fudging some arbitrary emissions standard. They are almost all still in business.

NOx emissions have probably killed people with smog sensitivities.

Probably, possibly, statistically contributing -- yeah maybe. But other automakers have made decisions that directly, indisputably, really killed people. Usually to save a couple of dollars on a part.

"...for whatever reason" is the part that got me.

Even if you look at it from a corporate politics standpoint, a software engineer will much sooner say "car's not passing emissions? Not my problem -- take it back to engineering".

Reply on news.ycombinator.com