Comment by Terretta | Hacker News Reader

Comment by Terretta | Hacker News Reader

Parent Full thread

Not comfortable with this "don't blame the engineers" angle, which sounds like some developers are taking the fall for execs.

We are talking about engineers, yes, but, guys like the research and development chiefs of Audi and Porsche, Ulrich Hackenberg and Wolfgang Hatz, or Hanno Jelden, head of the company's powertrain electronics.

These guys are top execs. Likely substantial performance incentive packages.

Such execs, like any CTO anywhere, could run a "special project" like this off the radar, to hit projected metrics, and bonus targets.

Don't take VW's word for it, here is the German prosecutor's take:

> The number of Volkswagen employees that orchestrated the diesel-emissions scandal that's rocked the automotive industry was likely fewer than 10, according to the public prosecutor's office in Braunschweig, Germany.

http://www.autoblog.com/2015/10/23/fewer-than-10-vw-engineer...

Often we engineers know things the CEO doesn't. I'm confident that happens across industries.

Over the course of my career 3 times someone wanted me to do something illegal. All of those times I handed in my walking papers. In one case I was asked to directly do something that would cause me to be acting in an illegal way, in two of the cases it would result in a company doing something that was illegal.

Yes, CTO's can run a 'special project' but the engineers are quite aware of the situation with respect to emission targets in this specific case and must have been very much aware that they were doing something that was fraudulent. The people mentioned there certainly had incentives but did not have VW in their pocket to such an extent that they could interpret their mandate as carte-blanche to commit fraud. These are not stupid people, on the contrary. My interpretation is that they were told to do this 'in whatever way possible' and that they interpreted this possibly as 'and that includes illegal but don't tell me about it'. Plausible deniability are the key words.

Whether that's the case or not will come out in due time, it always does. See Oliver North, Watergate and countless other scandals where you can only throw so many people under the bus until someone does a tell-all.

I note that the prosecutor hedges his words carefully ('likely' and 'orchestrated') and I also note that on the VW org chart it would not take more than 10 people to go from the CEO level all the way to a single programmer to insert this code.

That would also still require someone to either cooperate with this scheme at the auditors side, especially given the nature of the Bosch memo and the document linked by user PeterStuer which you may take as common knowledge within the automotive industry. In such circumstances a source code level audit of the software that ends up in the shipping devices to make sure everything is above board would seem to be indicated at a minimum.

That is the right way to handle the situation. These engineers work in such a specialized environment, I wonder how that would work out for them.

That's a good point, and it is likely that if they did not 'play ball' that they might have been thrown out. But in such a situation you could consider becoming a whistle blower. One thing is for sure, the moment you play along with a scheme like that when the chickens come home to roost you won't be able to deny you were involved and thus likely culpable. Some things are not worth any amount of compensation. Going to jail for some company or ending up holding the bag because everybody else pretends they have no idea why you did what you did would really suck.

Just imagine. Some German engineer was pressured into doing this. I guess it only counts when your decisions materially affect your own situation, in all other situations principles are free. Engineers should stick to their oath, and should use their fraternity to block stupid decisions such as these, even if that leads to individual hardship.

After all, what's next? Bridges that collapse? Buildings that you can't trust? Aicraft manufacturers? You have to draw the line somewhere and it is generally recognized that the engineering profession needs to be trusted by the rest of society if society is to continue to function. You can make mistakes as an engineer but you simply can not be caught even accessory to fraud, that's risking the reputation of the profession as a whole.

We also don't know how many, if any, engineers were asked to be a part of this, refused, and quit or were fired as a result. And possibly pressured in various ways to not say anything about what they were asked to do.

This happened in Germany. Believe me that no executive wants to fight with Bernd Osterloh and there isn't a single engineer who doesn't now how to fight back with the labour union as a backup.

Sorry, but whistleblowers, at least in the US, get screwed. Playing along or quitting is the smarter play; there's at least a chance you won't be badly hurt.

Examples abound. Do you know who is the only person sent to prison for waterboarding prisoners (and to be clear, torture)? The guy who told the press about it [1]. Hell, just yesterday there was a JPMorgan whistleblower on here who got fucked out of his job; they made sure he had 3 complaints on his record then lied under oath about who wrote the complaints so he can't find a new employer.

The upshot is, at least in the US, only fools report on their employers because there's a very real chance you lose your ability to earn a living. The anti-retaliation laws may be very pretty but appear to have zero teeth.

I wouldn't be surprised if the situation in Germany is similar. I'm not saying engineers should do this sort of stuff, but imagine for a second the costs to telling various governments what they were busy not seeing. So if the price of being honest is you have to quit one of the largest employers in Germany, who is quite possibly the sole employer of what you do where you live, it may well require you and your family to move. At minimum! Not to mention a backchannel negative reference if your boss feels like hurting you.

[1] https://en.wikipedia.org/wiki/John_Kiriakou

Germany is not the US.

Yes, whistleblower is not an easy path but there are many ways in which this could have been done without direct danger to the people involved (too many people knew about it to figure out who sent that anonymous package to the top 10 German press outlets). There are also some pretty powerful unions that could have helped with this.

And on the other side is the local government concerned about the largest local employer and the national government concerned about a systemically-important employer and industry.

I think (but hope otherwise!) that you're totally wrong about how the power structures in Germany would have reacted to this very unwelcome news.

> Engineers should stick to their oath

Um... what oath?

Germany has a professional society of engineers as well (the VDI).

It might not be perfectly obvious to everybody working on implementation that something is illegal. 'Our American lawyers tell us this is legally in compliance with EPA standards. Go forth and implement'. What are you going to say? You might have misgivings but you're probably also not an expert in regulations and their enforcement.

> It might not be perfectly obvious to everybody working on implementation that something is illegal.

Now you're calling them dumb. That's probably even worse than malicious. This is one of the reasons why I am bothered with the VW explanation they actively picture themselves as incompetent because it looks better on your resume than 'criminal' and they are not at all worried about throwing VW as a brand under the bus.

I'm not calling them dumb, just trying to point out that a scenario in which there is a very small number of people with direct and explicit knowledge of the cheating (or at a minimum, with no plausible deniability) is at least possible. An ECU is a big, complicated thing, much of it closely-guarded trade secret. So it's at least conceivable that it's developed in a compartmentalized way and that the bits and pieces that when combined enable cheating can be developed legitimately and independently.

There can be a legitimate reason to develop test cycle detection. And obviously, any ECU has the capability to put the engine in different regimes. Combining these two specifically for the purpose of cheating on emissions tests is not some major engineering project, if the pieces exist. In fact, you can make a decent argument that if you wanted to be maximally sneaky and evil, that's exactly how you'd do it. You'd keep the number of people without plausible deniability very, very low.

It has been noted elsewhere in this thread that Martin Winterkorn was 'hands-on to the point of being a control-freak' and that there was no way in which he would not have been aware of this. Note the amount of technical knowledge present in the VW boards and at the C level is such that they could have cut through all the secrecy and other red tape if they had wanted to. It would have also been impossible to hide this from the auditors if they had simply been of the 'trust but verify' mentality that one would expect of seasoned auditors and I find it hard to believe that a company like VW (which apparently has been caught cheating before when it comes to emissions testing) would be so incompetent at this that it could have gone on for so long. All this besides the fact that 'miracles' simply don't exist and if there is one thing that sets an auditor off it is 'too good to be true'.

An ECU is not a 'big complicated thing', there is a ton of software in a car but the ECU itself happens to be one of the more easily understood parts where anything that does not have an obvious and direct function stands out like a sore tooth. There is also a very distinct point in time where the problems magically disappeared and of course it would be a simple thing to review the changes committed around and prior to that time. (Let's at least assume they have some kind of source code control system.)

If anything the auditors are more trusted than the programmers and they would definitely have had insight into the ECU code.

It is not conceivable that there is some kind of 'the cube' like conspiracy where the parts each individually look clean because one fairly large chunk of code was contributed by the Bosch company and contained a fairly explicit piece of code to detect the test cycle. If it's labeled 'process bananas' and an auditor is not able to figure out what it does it is their duty to actually dig until they understand that piece, that is what they are paid for.

So I believe that you are principally right that besides one exec and a bunch of people down from there in the org chart they kept the number of people as limited as they could but there is no way the compliance department was this incompetent unless a rookie was specifically tasked with auditing that particular engine to make sure the cheating stayed hidden. A seasoned auditor would spot this in a heartbeat and no rookie should have been able to sign off on these changes on their own.

And 'plausible deniability' is a very very thin fig leaf away from being culpable, all it takes is a single email or confession to pierce that veil and with the number of people VW is throwing under the bus (and at ever higher levels) it is a matter of time before someone decides to come clean about the whole thing.

Well... I imagine we are more in violent agreement than anything else - I might have some quibbles with the notion that ECUs are that simple and, having done a bit of work in that industry in that part of the world, I have some trouble following the notion that 'compliance departments' are so rigorous and independent.

To me, VW's story is at least somewhat plausible because it is not difficult to intentionally and specifically set it up to be that way. I don't have any trouble believing that they deliberately went to great lengths to ensure that and that outside of a small group of crooks (culturally and otherwise encouraged and enabled from above), it was borderline enough that nobody stood up and said 'this is illegal and effed up, I'm going home'.

Fair enough. Given time we'll know it all anyway. Thanks for the exchange!

pvg is not calling them dumb. He is saying that they lack an understanding of the laws, regulations, and regulatory environments of different countries. And faced with this lack of understanding relied on their company and boss' judgement.

I don't consider myself an idiot but there are many times I read about actions taken by companies and go "S&*t that's legal!?!?!?". Many times unscrupulous and gray areas in the law aren't technically illegal. In addition the the Milgram experiments shows that even very smart people feel enormous pressure when when confronted with an authority figure. These two effects and I can imagine how hard it would be when the V.P. comes in and has a talk with you about how he had talked with legal counsel and they had looked over the regulatory rules and decided this was allowed, and other automotive companies were doing it.

The engineer could easily rationalize this away thinking "surely VW wouldn't do something illegal. They could get in so much trouble. I imagine I just misunderstood the laws."

The 'engineer' here is a C level exec, not some programmer stuck in a basement below VW's offices somewhere.

The best explanation so far of how this may have come to pass is that the compliance department and the engineers that actually wrote the code report to the same person but not the CEO. That would allow the CEO to claim he and the board didn't know anything about this. That still leaves you to wonder how they could have been duped so easily, after all 'tremendous problem solved by magic' normally does not pass the smell test.

If the engineers read the documentation of the code they were using it was explicitly spelled out what they were doing was a "bad idea". You're also arguing that automotive engineers don't have a basic idea on what emissions regulations exist.

As an aside, this same argument could also be used as justification for NSA programs that violate civil rights.

No.

When Kirk says "more power" and Scotty says "you cannot change the laws of physics", maybe Kirk should listen. But instead Scotty is forced to do something unsafe.

This is a metaphor for any engineering department given unrealistic performance targets. Engineers find solutions, so make sure they are targeted at the right problems.

Are you aware, that the german prosecutor you quote works across the hall of the biggest share owner of VW? Thats right, the ministry of the sub-state that owns most of the shares and which is not completely independent is what? Somebody here should be prosecuting his own boss.

It's not "don't blame the engineers", it's "don't shift all of the blame to the engineers". That doesn't mean some engineers weren't responsible, just that they don't deserve all the blame.

Reply on news.ycombinator.com