Volkswagen, Offering Amnesty, Asks Workers to Come Forward on Emissions Cheating | Hacker News Reader

Volkswagen, Offering Amnesty, Asks Workers to Come Forward on Emissions Cheating

nytimes.com

61 pointshvo68 comments

It's getting worse every day for VW it seems. First a limited range of diesel engines, then a whole bunch more, then the gasoline engines. Now this very transparent ploy at distancing. I hope they do realize that it isn't fooling anybody and that if nobody steps forward it doesn't prove a thing. They're in heap of trouble. It's interesting how those 'workers' had absolutely nothing to gain except continued employment from this whole saga and the VW management and stockholders had lots to gain.

If anything there should be whole slew of emails documenting this feature and its reasons for existence and who authorized this. VW 'asking' is tantamount to them saying they still don't know how this whole thing happened which is fairly incredible.

It's quite something to watch an industry giant flail like this, when Mercedes had their A-class debacle they got the story under control relatively quickly, VW seems to go from a certain level of bad to something even worse every time I read the news.

I think the key bit with this news is that they're doubling down on "the employees snuck this by us. We had no idea. Absolutely no clue whatsoever. Pinkie swear."

They may even be able to "find" an employee who has some kind of plausible story.

The reality is that most auto manufacturers cheat on emissions tests in europe. Mechanics who work on the cars know it. Even the regulators are hesitant to reform emissions testing.

The silver lining is that perhaps an actual improvement in emissions is happening, with the EPA adding road-testing to their emissions test suite.

>"the employees snuck this by us. We had no idea. Absolutely no clue whatsoever. Pinkie swear."

That's how I read the headline. Good thing I wasn't in mid sip when I read it[1]. You can imagine what would happen to an engineer that tried to bring that up. I swear you can smell that dynamic at a company within fifteen minutes of walking in the door.

[1] Seriously this sort of stuff happens, the board members should be banned from serving on a board of directors again.

> I think the key bit with this news is that they're doubling down on "the employees snuck this by us. We had no idea. Absolutely no clue whatsoever. Pinkie swear."

> They may even be able to "find" an employee who has some kind of plausible story.

Maybe! But then they are just claiming to be incompetent as an auto engineering organization, rather than malicious. An interesting choice.

It won't be long before their email servers are going to be impounded. I assume they've long since been scrubbed of evidence but then there are the backups. It's going to be very tough to keep that house of cards standing if it isn't grounded in fact.

Personally I wouldn't write a line of code like that even when authorized by management, I'd rather resign (but then again that's a argued from a position of relative luxury), but I hope that German programmers working for VW or Bosch would want to see some very explicit authorization that this is exactly what it is they are supposed to make with a note to the effect that they believe this is not legal or ethical or both.

If not they may very well end up holding the bag, I have no doubt that this thing will be researched to the bottom.

Not much different that what happened at GM except that it led directly to over a hundred deaths, yet what happens? Less than a billion dollar fine and a few people lost their jobs. The same claim you make of VW can be made here, but as usual the case involves far less than the top but instead the middle where someone likely needed to deliver something and did. How many were involved is the question and how they were able to keep it hidden will be interesting to find out.

Well, in contrast to GM, VW already suspended top managers, and the CEO instantly resigned.

That’s already a good sign.

> If anything there should be whole slew of emails documenting this feature and its reasons for existence and who authorized this.

Email trails are easy to get rid of.

My company rolled out full disk encryption using TPM due to HIPAA concerns. Then we got a new email retention policy of 6 months after which emails were deleted because we were migrating from Lotus Notes to Exchange. We had a local archive option for people who wanted to retain emails older than 6 months. Then we got a hardware refresh where those archives were not automatically migrated to new hardware. Not that the new machines had Lotus Notes anyways. The IT Asset group was instructed to remove Hard drives from old hardware and wipe them because HIPAA. Even if they hadn't wiped the drives, they weren't documenting which drive came from which machine when they binned them so you wouldn't be able to decrypt the drive because TPM. Also they were removing asset tags from decommissioned machines as they prepared them for the reseller. 12 months later they reevaluated the email retention policy and decided to increate it 24 months.

Emails? What emails.

Sure, but emails regarding possible illegal instructions from management would most likely end up being printed out and taken off-site. It also will look very bad if VW decided say last week to suddenly revamp their email infrastructure resulting in massive data-loss. If they had those emails a few weeks ago, they still have them. If they saw this coming long ago they may have found a plausible way to get rid of them.

Amnesty? Pressuring employees to "snitch"? Are you kidding me?

This is just blatant scapegoating. Your cars cheated on emissions test because you willfully ignored a decade of evidence! This is a case of pervasive organizational intent. You saw no evil, heard no evil, and spoke no evil, from the top management on down. Even the government regulators participated.

This is such bullshit.

Organizations are not people. They have a mind of their own, independent of individuals and yet controlled by the autopilot psychology of humans, unless systemically organized otherwise.

Things like this happen every day in organizations all over the world, when systems are not understood or willfully guided. It's just that most of them don't make the news—they simply silently eat at productivity and joy of work until a modern joyless inhuman organization is created. Everyone saw no evil, heard no evil, and spoke no evil—and it still happens.

It's not pervasive organizational intent—it's pervasive organizational chaos.

> Organizations are not people. They have a mind of their own, independent of individuals and yet controlled by the autopilot psychology of humans, unless systemically organized otherwise.

Exactly.

> It's not pervasive organizational intent—it's pervasive organizational chaos.

Well, in this case the chaos adapted rather optimally to the physics of diesel engines, the regulatory environment, and the emissions testing regime. I'd say it's worthwhile to treat that as intent -- to create incentive to decide how to act intentionally, rather than let chaos and circumstances form your intent.

Yes, Chaos as in the acceptance of the absence of systems to guide the whole in the correct direction. Chaos and circumstance formed the intent without any single person deciding; classic mob effect. There needs to be a system to prevent that type of thing, and it's often very complex.

Well, the CEO stepped down and multiple managers were suspended. New management might actually want to know, not find someone to blame.

Exactly. It’s useful to know who of your managers was responsible for what so you can use that in court against them.

At this point they seem to be simply trying to figure out how all this happened and went on for so long.

I don't think I'd take it. The article says you can still be transferred to a different department or given different duties, and they can't insulate you from criminal charges.

> At this point they seem to be simply trying to figure out how all this happened and went on for so long.

"I'm shocked, shocked to find that gambling is going on here!"

Your emissions testing reports, sir.

Shouldn't it be the US and German governments that are offering amnesty from criminal prosecution for Volkswagen whistleblowers?

Yeah. This seems like a calculated way for Volkswagen to make sure that nobody comes forward with anything, to "prove" that nothing else is happening.

I mean, they even say you may still face criminal charges for coming forward.

Though playing devils advocate here should not the civil servants and politicians who pushed diesel over petrol (and to hell with the known health risks) be in the dock.

I recall Mrs Thatcher queried this.

First, the US government has no power. Yes, they stole the passport of a VW manager[1], but VW since decided to instantly get all their people out of the US. But practically, since that, they have zero power.

Second, the German government is the largest external shareholder of VW[2] – so you can assume that this action was made under acceptance of the government.

[1] http://www.spiegel.de/wirtschaft/unternehmen/sueddeutsche-us...

[2] Well, the largest shareholder of VW is Porsche Group. The largest shareholder of that is VW. It’s complicated. Effectively, most of it belongs to the Porsche/Piech family and the German government.

I don’t understand what “Amnesty” would mean in this context. We are used to governments offering amnesty, but that is specifically amnesty from prosecution, and a normal citizen does not otherwise have much interation with the state in their daily lives. According to the article, VW workers are being offered amnesty from being “fired or [facing] damage claims”. However, there are very many ways in which VW could make workers’ lives a living hell without firing them, since employees by definition interact with VW all day, every day. VW would have to enumerate every one of those bad things they could do and promise not do it, which is, of course, impossible. This feels like an empty promise.

What amnesty means is clearly defined, as you quote from the article. I don't see why the amnesty offered by a company would be so different from that offered by the government. A government could similarly make your life unpleasant, just as a company, and in neither case is it customary to exhaustively enumerate things. Ultimately it is based on trust, and governments have also been known to renege on their promises. I'm not saying this promise is trustworthy though, just that the fact that it is from a company doesn't disqualify it outright.

As I said, an ordinary person has little interaction with the state, so there are fewer direct opportunities for harassment. Moreover, the state has little reason to harass the usual persons offered amnesty from prosecution, since those people have usually not been in direct opposition to the state itself – they are usually just criminals.

However, if they had been in opposition to the state itself, it would probably be a different story. For instance, I don’t think Snowden could ever fully trust a (theoretical) offer of amnesty from the U.S. government. A similar situation (to the VW one) exists with employees of the state who continue to be employed by the state – they interact with the state more (in its role as their employer), and more interaction means more possibilities of harassment. Also, even some people who quit their jobs would be affected; these would be people who were employed by state actors which have a lot of interactions with people in general, like the IRS, the DMV or the Police. I would imagine that former employees at any of these organizations who quit their jobs and turned whistleblowers would experience trouble from their former employers. On the other hand, a whistleblowing former clerk from, say, the Department of Veterans Affairs, would probably not.

What I’m saying here is that there are two factors: The degree of interaction a person has with the organization after the event, and the degree to which the person was percieved to be in direct opposition to the organization itself. The VW employees score very highly on both these factors, while the median person offered amnesty from prosecution from the state usually scores low on both.

Ah, I see your point.

There's no reason for anyone in upper management to trust this offer, but I don't see why VW would harass any normal employees coming forward over this. VW has a good relationship with its workers and a strong works council and they need their full support at this time.

If that were so obviously true as you make it out to be, VW wouldn’t have needed to even offer this “amnesty” to begin with – employees would have felt free to come forward without fear of being fired. Since VW did offer this “amnesty”, this is clearly not the case. Therefore, somewhat paradoxically, since VW have made the offer, it is an empty one.

Not quite. Before amnesty, we have a case where coming forward may meed confessing to being complicit in misbehavior, which could be rightfully punished. Amnesty says that that won't exercise that right.

No it doesn’t. Amnesty, in this case, is limited to amnesty from being “fired or [facing] damage claims”. Nothing else.

> VW has a good relationship with its workers

Such a good relationship that none of them felt safe coming forward in the first place AND they were the first ones VW threw under the bus...

This offer does not apply to upper management, depending on where your cutoff for "upper" is. From the article, "The offer applies only to workers who are covered by collective bargaining agreements; it excludes top management."

You got VolksWagen's initials backwards every single time.

Whoops; fixed, thanks!

Do they not have a source repository with commit history and code reviews with names attached? Is that something unreasonable to expect from a multi-billion dollar company?

The code was supposedly written by a vendor or subcontractor, so it could have been delivered in a "drop" to VW without commit history.

Also, the story seems to have been that the "cheat" was a special mode explicitly programmed for test purposes and not something totally under the covers. The "crime" was in enabling that mode under field conditions. That change may not be covered by source code revision control, if it were made in a config file outside of SCC.

In a field more of us here understand, it would be similar to coding a feature in a server that dumps user passwords to the log file in plain text. Such a feature would be useful for certain debugging situations, but it would be disastrous to have it enabled on a production box. So you'd put a big all-caps warning in the source code, documentation, example config files saying whatever you do don't turn this on in production. Supposedly the engine management developers did something along those lines, but the folks who built the cars ignored the warning and enabled the mode anyway.

That may be, but it ignores the fact that software was written for the express purpose of not just identifying if a test was being ran, but then taking action to change the results of the test has no real world use. Any case where you would want to change engine performance has no reason to be triggered only when testing. To get both included in production as-is and turned on requires quite a series of consecutive mistakes to be considered accidental.

> In a field more of us here understand, it would be similar to coding a feature in a server that dumps user passwords to the log file in plain text.

No, it's like coding that feature but explicitly making it turn off whenever it detects a security review. How is that justifiable?

Yeah, I like your continuation of my analogy. However, as I understand it, from someone I know who worked in the automotive sector a while back, all vehicles have a special emissions test mode. It exists for legitimate reasons such as you don't want to have ABS turned on when on a dyno for safety reasons. It probably also exists for less legitimate reasons arising out of the history of testing practice -- essentially that the test never has been truly representative of real road conditions, and everyone involved in the testing process knew this. That being the case, all you need then is to add one more flag to the list of "things to enable/disable under test conditions". I agree that somewhere there must be some kind of record of when and who made that configuration change though.

These details of emissions modes and engine control software configuration could easily explain the defeat.

What remains is explaining how engineering managers higher up (all the way to the top) could remain willfully ignorant of the fact that VW's diesel engines "passed" emissions without resorting to exotic tech like Mercedes' bluetec exhaust fluids. They knew there was some sort of magic in their engines that set them apart, but they either understood it was cheating, or they never asked any questions.

I suspect this will turn out to be due to some corporate mass psychosis similar to when an airliner suffers controlled flight into terrain because nobody wanted to disagree with the captain.

> What remains is explaining how engineering managers higher up (all the way to the top) could remain willfully ignorant of the fact

> What remains is explaining how engineering managers higher up (all the way to the top) could remain willfully ignorant of the fact that VW's diesel engines "passed" emissions without resorting to exotic tech like Mercedes' bluetec exhaust fluids.

How often do high level managers at a big company have a good handle on technical implementation details of specific systems and what the implications of those details actually are?

It's possible that these people either had non-technical backgrounds, technical backgrounds in unrelated areas, or out of date technical backgrounds (like they last worked with this stuff back when cars still commonly had carburetors).

Martin Winterkorn (the VW CEO who resigned) was an engineer. Specifically, he was an powerplant engineer who helped spearhead VW's diesel technology.

I wouldn't be surprised to see him go to jail.

Okay, I can accept there may be a legitimate reason to want to determine if a test is being run for a case like that, but that's a component that isn't related to engine performance and emissions. I'm not sure there's any case where you could legitimately want the engine performance and emissions to be different during emissions testing. In any case you want it different (for example, testing different parameters), you would also want to test it on the road.

> That may be, but it ignores the fact that software was written for the express purpose of not just identifying if a test was being ran, but then taking action to change the results of the test has no real world use.

I think it depends on how general the implementation was. I could imagine real world uses for a feature to generically detect certain conditions and change parameters in response (e.g. to support different settings for highway vs city driving). If you have something general enough, then the "defeat device" could have just been a ruleset that only matched a very narrow set of conditions that usually only occurs during the tests.

My point is I'm not sure of any scenario which would need that criteria to me met, and the chance that it was accidentally implemented such that it just happened to match test conditions and not other conditions, and just happens to enable running conditions that are sub-optimal for operation but allow it to pass emissions seems so vanishingly small that I can't consider it a possibility without further information coming to light to explain why it would make sense.

Incompetence happens, but requiring chained events of successive incompetence to explain an outcome to explain an event doesn't seem rational to me.

Oh, I think that however it was done, it was done intentionally. My point was only that the "code" could have been kosher and the tampering done later via configuration.

Interesting. If this is the story, then VW didn't really "need" to cheat to make the cars reasonable, so they shouldn't get too much worse when patched. That would be a nice outcome.

I'm still cynical.

I don't understand how that follows. From all I've heard, people suspect that the difference between cheating and test mode is large.

> Such a feature would be useful for certain debugging situations, but it would be disastrous to have it enabled on a production box.

I've seen that exact thing (and worse...) on a production server in some job recently. But that big all-caps warning wasn't there, the docs were absent and there was no config bit to switch on/off, I found the file more or less by accident (wondering what all the io was that ate up the machine).

Agreed, but this has to be more than just software. The engineers must have some idea of emissions then suddenly it's not a problem any more? And they just think "that's lucky".

This is exactly the use case for signed commits and pushes.

VW != Silicon Valley startup.

He did say code reviews

So this German company's management says it was those lousy employees that did it and the employees say they were just following orders... wait, why'd my Godwin meter just peg?

No, the management was completely suspended. 10 board members and 2 CEOs were suspended or resigned over this.

The new CEO and board just want to collect evidence whom of the board gave the order (and thereby can be sued in court for damages)

I hope they paste these amnesty notices in the board room and exec offices.

Oh give me a break. Company makes the money, company takes the responsibility. Not the employees.

The best way for VW to come "clean" (sort of speak) out of this is to promise to everyone, including the governments, that they will be replacing all of their new models with EVs by 2025 (at least the street cars).

Well, they already promised (a) to refit all the cars without performance losses, (b) pay all fees and fines the car owners might face, (c) offer every single of their models as electric by 2020.

Now, if the German government is going to make the cars exempt from the 0.26 € per kWh electricity tax, they will be cheaper than gas, too.

Lying to a federal official can land you in prison. (Ask Martha Stewart)

Someone programmed these cars to 'lie' to the Feds. This caused damage to the environment and these cars dropped in resale value as a result, erasing millions of dollars of value from personal balance sheets.

Someone (not the shareholders paying a fine) should be held personally responsible and face criminal prosecution.

That’s exactly why 2 CEOs resigned and 10 top managers were suspended.

This amnesty seems to be – from what I get from the media – something the new CEO wants to be able to collect evidence against his predecessors.

The new CEO wants to collect evidence? The new CEO is a puppet for the board of directors. The CEO doesn't care, he's just there as a public face to play pretend that the company cares.

Either management is behind this, or they created the incentive for it to happen. They can't escape their responsibility; nobody would do it without a reason.

I think one of the biggest tells would be if the VW cars with the software issues weren't equipped well enough to deal with the emissions regulations. In other words, if the cars provably did not have the right equipment to meet emissions requirements, AND had the software issues, I think this would prove beyond reasonable doubt that VW was maliciously trying to circumvent the tests.

I see Chevy when advertising their new diesel Colorado put great emphasis on the validity of the testing it underwent.

I'm still at a bit of a loss on this whole rogue employee scenario. I'm a little more concerned about the rest of the software if this is true.

I don't get it. How many people could possibly have worked on that software? Surely less than a dozen. Why doesn't someone create a throw away email account and send an email? Better yet, why didn't they before the software shipped?

It must be a witch hunt at VW right now...

Not that any witches exist of course, just for a scapegoat.

I am impressed.

Reply on news.ycombinator.com