Hacker News Reader

Hacker News

Comment by im3w1l | Hacker News Reader

Parent Full thread

im3w1l

Many important differences.

1. Actively detecting test and behaving differently. It's like stealing a test vs teaching to the test.

2. Lower stakes. Health issues are much more serious than inefficiency.

3. It affects the buyer. It's more acceptable for the buyer to be cheated than everyone around them.

4. People could have created these layers by accident. Favouring those who got lucky is unfair.

Honestly I think basically all my gadgets exaggerate how energy efficient they are, by tuning parameters for tests that don't correspond to the real world. My dishwasher has an energy efficient mode, the manual literally says it's just for compliance and recommends other modes. It's just a fact of life.

View on HN

This is omnipresent even where regulators aren't involved: every graphics card benchmark out there is 'manipulated' relative to real world performance. At this point it's so universal that I don't think anyone is even fighting it - as long as everyone games benchmarks roughly the same amount, the relative scores stay usable.

Your point about fairness and passive design is the one that makes me view these cases differently also. In the anecdote, the product being tested was the same one being sold, and there's no sign the heater was worsened to improve test performance. The designers just picked the best-scoring option among some reasonable configurations. (Frankly, once they noticed that issue, what were they supposed to do? Pick the worst-scoring, or pick the spec out of a hat?)

In the VW story, the test-bench vehicle was fundamentally different from the market vehicle, and the road version was designed to behave worse on the metrics to get other gains. I happen to know someone who bought a diesel Jetta specifically because it was more eco-friendly than other options, and I think he'd draw a clear line between tuning for test metrics and VW consciously lying to their buyers.

It's interesting you mention graphics cards because that very behavior has lead to the gaming community favoring benchmarks derived from a handful of current gen games min/max/avg FPS over so called synthetic benchmarks. It only took a handful of instances of companies baking in "benchmark" modes that get triggered when certain benchmarks are detected for people to start discounting those benchmarks in favor of more organic measurements.

> Honestly I think basically all my gadgets exaggerate how energy efficient they are, by tuning parameters for tests that don't correspond to the real world.

Having measured all my gadgets with a Kill a Watt meter, that's not my experience. It seems that many gadget-makers realize that people don't really care about power draw, so they just slap the maximum draw onto the specs.

Reply on news.ycombinator.com