Hacker News Reader

Comment by JoeAltmaier | Hacker News Reader

The issue appears to be, that lab testing does not emulate real-world driving conditions. So in the field, emissions are higher than in the lab.

These four car makers involved in the report differ from VW: they do not have a defeat device in their design. Their results are solely because of the lab-vs-road issue.

Its not clear this is any kind of problem. Reducing emissions in lab tests may very well result in car designs that have reduced road emissions. That was the hope when lab standards were introduced. Auto pollution in our cities has indeed gone down as (lab) emissions standards have been tightened over the years.

View on HN

Its not clear this is any kind of problem. Reducing emissions in lab tests may very well result in car designs that have reduced road emissions.

This is the logic that European politicians have used in the past.

If there were a direct relationship between lab test emissions and real-world emissions, that approach may be acceptable. Unfortunately, that is not the case.

In the case of NOx, implementing the Euro standards should have resulted in an 80% reduction in emissions over the past 15 years. As the vehicle fleet is renewed, that should have translated into significant reductions in kerbside NOx levels. But this has not happened, and London continues to suffer some of the worst NOx levels in the world.

It turns out that an 80% reduction in lab test emissions has not translated into anything like an 80% reduction in real-world emissions - not even close!

The current system is inadequate because manufacturers simply "design for the test". There is huge real-world variability between vehicles that may record very similar lab test scores. A vehicle may claim similar test score results as another, but in fact have hugely differing levels of emissions in the real world.

Test environments need to reflect real-world conditions!

There is a new testing procedure (WLTP, Worldwide harmonized Light vehicles Test Procedures). Major carmakers just want to delay it:

http://www.transportenvironment.org/news/carmakers-pushing-3...

The emission cycle used now (New European Driving Cycle, or NEDC) is actually a joke, nobody drives like that: https://en.wikipedia.org/wiki/New_European_Driving_Cycle#/me...

comparing to the new one (WLTP): https://en.wikipedia.org/wiki/Worldwide_harmonized_Light_veh...

With the new testing procedure in place, you can still cheat like Volkswagen. But it will benefit the customers more, when the OEMs try to game the system by optimizing solely for the cycle.

What they need to do is more like a clinical trial for a drug.

    Step 0) measure on a dyno
    Step 1) measure on small closed course with a handful of drivers
         who are unknown to the car companies
    Step 2) measure on a larger group of cars in the real world

If these "costs" are unacceptable, then auto manufacturers shouldn't be putting cars on the road. I always thought trusting the ECU to tell the truth was ridiculous, mostly because over end user tampering not the original code. But the effect is the same.

It really is bullshit that so much time has been spent overfitting to the test when real universe exists right here. No model required.

It makes technically no sense if you can harmonize the regular driving condition into one single realistic cycle. Then the results are reproducible and comparable.

OEMs all have their in-house driving cycle to better reflect the reality[1] than the boring standard cycles.

The emission/consumption performance for some given customers is easily predictable.

To prevent cheating, you'll just have to open source the software inside.

[1] Lino Guzzella, Antonio Sciarretta: Vehicle Propulsion Systems: Introduction to Modeling and Optimization (page 22) https://books.google.de/books?id=aVNEAAAAQBAJ&pg=PA22&lpg=PA...

My assertion is that you can't do that with a single realistic cycle precisely because there is a single one. Overfitting will occur, an ensemble of actors can't be gamed.

Sort of and sort of not right? So VW came right out and said "The difference is caused by defeat device code programmed into the computer." Great simple answer. None of these people have offered up a theory of why things are different. It could be defeat devices, or it could be something else, they just don't know.

So step two of this process is figure out all the ways in which actual driving differs from lab testing. And when you've eliminated all other explanations, start looking for defeat devices or their equivalents.

I'm guessing one or more of these companies will end up admitting, after looking into it, that their car control software runs differently if it detects the car is being tested.

Maybe we have a different idea about what a 'defeat device' is. Its a physical piece of hardware. Not some code. So to find out if these other cars have it, you stick your head under that car and look.

The term "defeat device" refers to any mechanism, physical or otherwise, designed to defeat emmissions controls. It's not literally a standalone gadget. I assume the term is a holdover from the 1970s.

    A defeat device is an AECD “that reduces the effectiveness of the emission
    control system under conditions which may reasonably be expected to be
    encountered in normal vehicle operation and use. Motor vehicles equipped
    with defeat devices cannot be certified.”

And the AECD is defined as

    any element of design which senses temperature, vehicle speed,
    engine RPM, transmission gear, manifold vacuum or any other parameter for
    the purpose of activating, modulating, delaying or deactivating the
    operation of any part of the emission control system.

http://www.autocar.co.uk/car-news/industry/vw-emissions-scan...

What physical defeat device is VW using? (There isn't one)

Two temperature-sensing devices, according to the report.

Link to said report? Temperature sensors (EGT) are a normal part of cars.

That's an extremely narrow and frankly useless definition...

At least in my corner of the industry (commercial) , we've been asking for better test cycles because we don't want to wind up in a situation where we are chasing fuel economy on the test cycle at the expense of fuel economy in the real world. The regulators want to keep the same test cycles so they can measure how emissions and fuel economy are changing over time.

Internally, we have much better test cycles correlated to real world routes that we use to validate fuel economy simulations.

I don't understand why regulators don't just do additional, more representative tests so they can track long-term trends and also have a better test? How much time/resources do the tests actually take?

A more representative test (like running a car on a treadmill) requires a dedicated testing facility that nobody wants to pay for or wait in line to use.

The article suggests on average real-world emissions are 2-5x more than in the lab except for a 20x outlier. VW was 40 times more than in the lab.

I don't think most of these cars are hiding anything, it's just an example of how easy it is to optimise for a fixed test.

exactly, beside, a disconnect between test regime and real world results is expected.

what VW did was not just optimize for the test, but cheat the test, which is: those cars in the article are able to pass the tested restriction in the test regime, while disabling the cheating code wv engines were found not to be able to match test restriction even under test regime.

no one here was busted for real world emission. that's kind of understood already by regulators which built the test assuming the result would trickle down in real world usage even if not 1:1

It's not as though there haven't been issues with gaming these tests to a degree for a long time. Almost certainly this has been an issue of walking backwards into a big mistake little by little, which is the typical process of making big mistakes. You start by figuring out how to game the system right at "the line" as much as you can, then the day comes where you decide to be more explicit about it and build it into the software. You probably decide that you just want to play around at the edges, with incremental differences between "cheat mode" and real driving in the same realm as the bit by bit gaming they've been doing already. But once the system exists the natural process of optimization would push it to the limit of what was capable, and it turned out that was a lot. But by then you're committed, and it would take a lot of fortitude to walk back from such a mistake.

I'm sure the same sorts of dynamics have played out in many other auto makers, though it's notable that some, with a more developed sense of ethics seemingly, have not gone down that road.

"Its not clear this [lab-vs-road discrepancy] is any kind of problem."

I agree with what you said, except this.

Clearly these NOx emissions are a problem, because they cause smog (like ozone), which is a health hazard. The city where I live has too much ozone for many days of the year, for example.

Unless this problem can be controlled, vehicles like these should not be sold here, and people will have to find other options, or technologists will have to develop better cars.

Except that's not what he was saying. It's been known forever that stated milage and actual mileage differ because the lab test is in a lab.

If people want more accurate tests, then the lab test needs to be modified to be realistic.

My point is that this discrepancy has, in the real world, proved to be a huge public health problem. And we are not talking about mileage. We're talking about NOx that produces O3.

> These four car makers involved in the report differ from VW: they do not have a defeat device in their design. Their results are solely because of the lab-vs-road issue.

At the end of the day though, all the automakers did whatever was required to make them pass the lab testing, with no regard whatsoever for what that actually means on the road.

Maybe it's good on the road, maybe it's not, but in all honesty none of them care.

VW went about it different than the rest of them (that we know about), but the end result is the same - cars that pass in the lab and do who-knows-what on the road.

That's not necessarily unreasonable. They complied with the regulation, which is determined by lab test. Perhaps the test is defective, but what else were they supposed to do?

VW acted in bad faith and clearly exhibited the mentality you describe, but it doesn't seem to me that we know enough to determine how these few acted.

It does seem apparent the test is fairly useless. If we want to pursue this type of emissions control, everything I've read suggests that it's very possible to hit the marks in real-world conditions at the expense of power and fuel economy.

I agree. It's wildly accepted the winner in most sports contents is the one the pushes the rules the hardest - i.e. push the line as hard as possible, without "breaking" the rules.

I've not read the EPA rules, but I'm sure it says something like x amount of y gas under whatever conditions.

At the end of the day, VW meets those rules.

They definitely didn't. EPA didn't just make up the term "defeat device" last month. It's in the regulation that VW was supposed to comply with.

The EPA rules also contain a NTE "not-to-exceed" limit, about 2.5x the test limit. They don't have much of an enforcement budget, so testing is pretty rare, but if the car exceeds the NTE limit, the manufacturer has to issue a recall to fix it.

Well, they pretty clearly f'd up by designing tests that differ so hard from actual usage. Given that you only have to sample a small set of the cars, it shouldn't be that big a deal to have it go through longer, more representative one, even if it allows greater intra-test variation. (Perhaps compensate by weakening the standard, since it's more accurate that way.)

>These four car makers involved in the report differ from VW: they do not have a defeat device in their design.

I think it's a difference of degree, not kind. They all optimize narrowly for one test at the expense of real-world usage. VW just optimized a lot more narrowly.

It's like compilers. There is optimization by recognizing cases that might occur in programs. And then there is optimization by recognizing ... entire functions featured in specific benchmark programs, or those entire programs themselves.

Imagine a compiler that recognizes when you're attempting to compile Firefox from source. It checks your internet connection speed and hashes all the source files. If your Internet connection is sufficient, and the hashes match, this nifty compiler would "optimize" the compilation by downloading the matching binary from mozilla.org.

ccache hashes the preprocessed input to gcc, hashes it, and pulls out a previously compiled object file if possible. You can use it together with distcc to farm out jobs to other machines; whether you pick their caches or yours depends on the order of ccache and distcc on the command line.

So basically any caching strategy?

Compiler optimisation isnt about compiling faster, though :-)

Of course it is, at least if the optimized compiler is compiled with an optimized compiler, but also generally fast compilation is optimal.

Fast compilation is a worth goal, but it's not fundamentally different from optimizing other kinds of programs: it involves good selection of data structures and algorithms, smart caching, that sort of stuff.

Compiler optimization (in the sense of optimizing the emitted code) is a research area of its own with millions of man hours invested.

Oh my. I've worked in those car test and simulation SW environments (and a cross compiler for a weird engine), and now I'm working on compilers and compiler optimizations in a different industry.

You cannot compare that at all.

With engine simulation and control you have real-time loops, with the biggest problem being the latency. You reduce latency by getting better HW, better sensors, shorter distances, a better SW driver architecture. You don't optimize by finding patterns or counting occurrences at all, esp. not at run-time. You always have to be prepared for the worst case. Plus you always have HW + SW combined. Any major important SW feature is always doubled in HW for security. It's a different world.

With compilers there's no real-time trouble (GC's, ha!), mostly no loops, no worry about HW, cables, sensors or drivers. It's pure abstraction. With JIT compilers it's a bit different, mostly a tradeoff memory + optimization overhead vs speed. More measurements and dynamic decisions. But jit optimization in engine control? Nah. And there's never a HW cross check. Signals, interrupts may arrive or get lost. The machine will not hard reset and reboot if a signal gets lost. It does so in a car or in space or other industrial control environments.

The similarity is that optimization is taking place, and cheating is possible by recognizing a test situation, in order to "game" the test. Compilers can "game" benchmarks by recognizing them specifically and supplying hand-crafted code. The cases which do this have no applicability to non-benchmark code. Just like recognizing that an emissions test is taking place and behaving optimally for that test has no applicability to real driving.

Since I worked on this SW I have to clarify a bit.

Lab testing of course does emulate real-world driving conditions. Otherwise your tests will not be worth much. Either you play back a recorded existing course (RPS - road profile sequences) or you simulate it, simple steady state and transient cycles. We are automating reproducible test conditions.

But we are here only talking about the different emission tests, where we have 2 major philosophies. The european emission test tracks (NECD), which is also used in India and Asia, which is an highly abstract test with the engine alone driven by a similator simulating multiple real-world driving conditions (a few urban cycles) and the american test on a dyno with a real driver (with varying cycles), so that the american car manifacturers, who traditionally were not so good in designing emission-free engines could traditionally cheat by hiring the best drivers to pass the test.

So Americans have cheated historically the most, but apparently engine manifacturers which couldn't catch up with modern diesel engine development had to cheat also. First it is trivial to cheat, since the test has to be done only once. This is entirely a political game to get the NOx and COx values down, but don't harm their own industries. And esp. Americans seldom use scientifically based values.

And then you have cold starts in reality, which is mostly not tested in the emission cycles tests. Emissions will always be higher in the field than with a properly warmed up engine under lab conditions. So it's no surprise that different measurement methods will lead to different results.

Reply on news.ycombinator.com