How Samsung inflated its performance scores | Hacker News Reader

How Samsung inflated its performance scores

theguardian.com

52 pointsantimora30 comments

"We've already started work on making sure that all future versions of benchmarks we get will come with unique package names." - Anandtech

After that Samsung will just figure out some other identifying feature to recognize the benchmarks. Naturally we might go overboard and do random obfuscation on everything, that might solve the issue for the journalists who use them one off.

However for benchmarks that are published in the Google play store that is impossible, so there is no real way for benchmark developers to 'immunize' their products for this sort of approach.

Disclaimer: I am employed in developing Mobile benchmarks, and our product is something they cheat on. Avoiding devices that cheat is very much in our interest.

> After that Samsung will just figure out some other identifying feature to recognize the benchmarks.

Yep, reminds me of 3DMark2003 when NVidia and ATi "optimized" their drivers by disabling, altering or swapping the benchmark's shaders for their own or less complex versions

> The phone giant has proved that benchmarking apps are vulnerable

The only thing they have proved is their willingness to game benchmarks. ATI and NVidia were gaming 3DMark 10 years ago.

Also interesting to note, though Samsung is definitely more than willing to game benchmark despite their protests (and looks like the worst offender by a pretty long shot), Anandtech found out they're far from the only company doing so: http://anandtech.com/show/7384/state-of-cheating-in-android-...

Edit: turns out this was mentioned in the article, but the way it's presented, what the fuck Guardian?

> Anandtech, a site where you could get the (wrong) impression that benchmarks are the only reason to own a gadget

Anandtech's Article:

Cheaters: Samsung, LG, Asus, HTC

non-cheaters: Motorola, Google, Nvidia, Apple

See any patterns?

Let me guess: Suppliers of commodity hardware have an incentive to make that hardware look as attractive as possible, suppliers competing on other dimensions lie about other things ("We're open!, "It's magical")

Or did you mean something else?

Not really, care to enlighten me?

  Cheaters: Samsung, LG, Asus, HTC

Korean companies

  non-cheaters: Motorola, Google, Nvidia, Apple

US companies.

I for one can't help but see a slightly xenophobic pattern in the poster's comment.

ASUS and HTC are Taiwanese, not Korean.

I stand corrected.

Thanks

Still... Korea+Taiwan vs US.

This was a list of companies provided by Anandtech in its reporting. I just found it amusing that it happened to be a pattern in this set of data. Apologies!!

Perfectly correct. The only way to produce real benchmarks is to run useful software that are identical on two different hardware setups. Sadly because of to DRM, this is impossible to do on lock down devices.

You could run two similar applications (say angry avialaes), but the only thing you would test is that device implementation of said game.

The funny thing is that despite being caught out doing this a while ago with the S4, they have no choice but to continue doing it regardless how embarrassing it becomes. If they don't, their next generation devices may score only slightly more (or less than) than the current gen devices.

They painted themselves into a corner from which there is no escape. And they only have themselves to blame.

But it's not like the market punished them at all. They are still selling lots of them.

Apple is the only that held them accountable in court, at least a little it. These b*stards are shameless http://cdn.mactrast.com/wp-content/uploads/2011/09/Samsung-A...

Your proof that "Samsung are bastards" is because they have a connector that is either a PDMI connector, or something that looks suspiciously similar?

While I don't agree with the assertion that "Samsung are bastards," they very obviously used a PDMI connector because it looked more like Apple's dock connector at the time.

Not only did they not use micro-USB like most manufacturers, I recall reading that their PDMI implementation was non-standard (i.e., not-quite-PDMI). There was no reason for them to choose such an incompatible connector other than the shape resembled an Apple connector.

How is this standard called: http://media.idownloadblog.com/wp-content/uploads/2011/09/Sa... ?

A cube?

People still buys. In business sense, it is funny. They cheated and impressed as much as they can and made really large volume of sale. Media reporting like this does not seem to make much difference. Other than few hackers/engineers/programmers, majority of consumers don't give a thing on benchmarks. Too bad.

The CPU/GPUs keep getting faster but this idiotic PPI race is forcing manufactures to cheat so they can appear faster than the previous generation. On a 5" display anything over 720p is overkill.

A number of benchmarks have an "off-screen" mode with fixed-resolution rendering specifically to avoid native resolution artefacts. That provides both the absolute hardware power in comparison to other devices and the device's "feeling", how its hardware meshes together (aka if it has enough power to drive its own screen). The second one is arguably the more important one wrt user experience and the ability to actually use the device.

GLBenchmark has both onscreen and offscreen modes, and Anandtech posts both every time.

If Apple moves to a 5" screen or something close next year, I'm sure they'll double up the resolution again to 2272x1280. That could force Android OEM's to think they need to go to 2560x1440 now, to be higher than Apple, but I agree that would be silly.

It could help devices like Oculus Rift, though, and it could also lead to 4k displays on 10-12" tablets and notebooks faster than otherwise ("only" 2x the pixels from the current resolutions for 10" tablets).

All I can think is, imagine if Apple did this.

Samsung is one shady company. They get away with murder in South Korea, given that they make up a huge chunk of SK's GDP.

http://en.wikipedia.org/wiki/Lee_Kun-hee#Samsung_scandal

They do make decent electronics, but I wouldn't trust anything they say.

So the chairman was found guilty of financial wrongdoing and tax evasion.

Tax evasion is indeed serious business, but murder? I guess I would like to see several large business CEO's charged with facilitating murder by tax evasion. Would make for some interesting times, and likely tax cuts for the general public.

To "get away with murder" is a very common way of saying someone is successfully avoiding the consequences of their actions.

Concerning that Samsung operates like this even though they are the dominant market leader - with a plurality of the market share for mobile devices, 30.4% of the global market Q2, 2013 [1]. 1 http://bgr.com/2013/07/26/smartphone-market-share-q2-2013-sa...

I think Samsung recognizes that their position as a "dominant market leader" is due more to luck than anything else, and that their products really aren't very different from those of their main competitors.

They have good reason to be nervous, maybe even more so than their competitors: Samsung gains a lot from the public perception that they're "in the lead", and thus have more to lose should that perception change...

Well, their washing machines are OK. Anything computer or phone related is cheap rubbish for those with no money or with low standards.

I've been an iPhone user since the first one shipped, and just changed to a Galaxy Note 3. It's neither cheap nor rubbish. But I do feel a bit uneasy about supporting them in light of their history of "extreme homage" in design. I'm not at all surprised to hear about them acting shady in this way.

Reply on news.ycombinator.com