Note 3’s benchmarking “adjustments” inflate scores by up to 20%
arstechnica.com
arstechnica.com
> The ironic thing is, even with the benchmark booster disabled, the Note 3 still comes out faster than the G2 in this test. If the intent behind the boosting was simply to ensure that the Note 3 came out ahead in the benchmark race, it doesn't appear to have been necessary in the first place.
The Samsung phone division upper management and marketing team don't seem burdened by an overabundance of neither ethics nor common sense. Or maybe I'm just being naive.
Renaming quake3.exe to quack3.exe would fool the detectors and skip these "optimizations".[1]
[1] http://www.hardocp.com/article/2001/10/23/optimizing_or_chea...
Customers who actually want framerate-beyond-all-else don't enable/would disable those settings in the first place.
So if it were "giving the customer what they want" they'd have respected the customer's settings.
What it actually was, was a simple recognition that people used "framerate at max quality" as a proxy for net card performance and an attempt to "game" such measures by not actually running at max quality, for those games that were common benchmarks.
Do whatever the fuck you want, but be upfront and transparent about it. If you aren't, you're shady and sleazy.
Samsung is at it again. Doesn't matter if it's already a fantastic device, they still pull this kind of stuff.
sources: [1] http://www.rediff.com/business/slide-show/slide-show-1-tech-...
>"the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off."
Good! I want to know what the maximum performance is from a benchmarking app.
I don't know why regulars apps are unable to get the Note 3 to use all 4 cores. Perhaps they're not multi-threaded or they haven't enabled the right settings? Surely 3D games will access all cores?
At the end of the day, Samsung wants the benchmarking to show the theoretical maximum. Other manufacturers may not be too bothered and are happy to show benchmarking of normal usage. It's like reading the specs on a car, you're never going to drive at 160 mph, but at least you know you could.
Saying I want to see the maximum performance is fine but they should also be forced to show how that impacts other elements of system performance.
Personally I think it would be good to see two sets of benchmark typical and maximum, but the most important thing is that whatever scores are being shown are clear and honest and I think it's hard to say that these are.
The way this probably should work is that the benchmarking apps themselves have a flag telling the OS they want all cores at full power all of the time, instead of having to hardcode this list into the OS. (Perhaps this could be one of the permissions you have to accept when installing the app.)
As I said, I don't mind if they put it flat out for the benchmarking, so long as they do the same for the battery tests too.
looks at a mobile phone with a 2 hour battery life
Your car spec analogy is a good one. If the car maker removed the seats and carpet as well as inessential engine components before testing, that would be the equivalent of what Samsung have done here.
It would also be fraud.
And they should be vilified for re-interpreting preconditions that would usually guarantee valid relative comparisons and are used like so. ("Theoretical maximum" is a problematic end to work towards, though - why shouldn't clock-speed specifications suffice, then? I presume you're talking about stock speeds sans the software stack - but cranking up frequency scaling isn't much of a meaningful hop and skip away from just running the benchmark.)
Many cars have "race" modes which allow the computer to use the maximum power an engine has while not trying to force it into an "efficient" mode.
99% of the time you want your car to be as efficient as possible while others you want to see what it's capable of.
The Nissan GT-R does this automatically when you're located on a track:
http://www.engadget.com/2007/12/22/nissan-gt-r-disables-spee...
It's like testdriving your car on the manufacturer's track, seeing it go to 160 mph, then taking the car to race day and you hit a 90 mph limiter they built into the car for fuel efficiency.
Ironically, there's a Nissan car that does enable and disable the limiter based on the GPS detecting it being on a race track or not. But they're transparent about it.
It's the GT-R, and it's only on the Japanese version: [trigger warning: gadget blog] http://www.engadget.com/2007/12/22/nissan-gt-r-disables-spee...
After all when Nvidia 6xx and 7xx series self overclock when there is room in the power envelope available this was disclosed and taken into account by the reviewers.
That doesn't explain why the exact same app performed worse (or normal, rather) when renamed .
The customer is buying a complete product, not a cpu. Maximum cpu performance without context is not meaningful. A 500 horsepower engine is pretty awesome, until you drop it into a cargo freighter. Then it's worthless.
Somewhere along the way they became a huge global company that makes a lot of money, but they forgot that they need to start acting like one.
They don't need to fake benchmarks or anything like this. They are the main event when it comes to Android.
Samsung is who we thought they were: http://www.youtube.com/watch?v=kw5UWqpcsDs
Ironically, I believe these smartphone OEM's are a lot less bad than PC OEM's/chip makers. I hate how Intel, Nvidia and AMD all "win benchmarks" or show numbers much higher than the practical limit of those chips.
All of them seem to have a "base clock speed", and then a "turbo speed" that goes even up to twice as high. What do you think gets shown in the benchmarks? The base clock speed, that you'll be using most of the time? Of course not. It's the "turbo" burst speed that will be calculated by the benchmarks.
This ends up being very misleading because for example I bought a laptop with an Nvidia GPU that was supposedly 3x faster than the integrated Intel one. But after about 30 minutes of playing a game, it will invariably get throttled to about half of that speed, and then it will only be 50 percent faster than the integrated one.
Intel is even worse than this, because they're becoming more desperate about the ARM competition, they start to become increasingly more misleading to make the appearance that they're "closing the gap" with ARM much faster than they really are.
Before I start, I want to say that I use a Core i7 in my laptop, but I just hate shameless misleading of consumers, whether it's from Intel, Nvidia, AMD or Samsung.
There are 3 "misleading strategies" that Intel has used lately:
1) Turbo-boost, of course. Their new Atom chips are actually 1.3-1.5 Ghz, but they market them as 2-2.4 Ghz chips, a speed that also helps them do very well in CPU benchmarks (they're still about 2x behind any other ARM GPU, but I think they'll start using turbo speed for the Atom GPU's in the next generation, too, again to make it "seem" like they caught up - except not really).
2) Launching the most powerful version they have first, even if it's not going to be used in most of the devices the consumers want, but this way the press is all over it, and then the consumers remember that "brand" that was very powerful, even if it's not what they're getting in their device.
Example: Haswell's Iris Pro. It was the first to be sent to sites and benchmark, and it showed much bigger GPU performance than before in laptops. The only problem? It's probably never going to be in a laptop. It was meant for PC's mostly, and again as a "halo product", to make people think that "Haswell is very powerful". In fact most laptops have only seen about 20 percent increase in GPU performance with Haswell, compared to IVB.
Second example: They launched Bay Trail first, the "tablet version", to make people think "how powerful Atom is" and that it's "good for smartphones", and then they will release Merrified for smartphones, half a year later (which would already put them behind competition), and probably even weaker than Bay Trail performance wise. But most consumers will just remember that Bay Trail was great in benchmarks (see 1, also), even that's not what they're getting.
3) They're starting to push the "SDP" measuring, so tech websites can start writing about and promoting their chips as "5W chips" or whatever, and hope that even if they're now using the SDP name after the power consumption number, they will eventually forget to mention it ("Intel's 5W chips", etc).
SDP is nothing but an absolute shameless scam from Intel. They want to have their cake and eat it, too: a) declare super high performance with the burst speed that it's not practical at all for too long, and b) declare super low-power with SDP, even if the chips themselves can get a lot hotter than Intel is implying with SDP, especially if they start using the Turbo-Boost speed.
These guys explain it a lot better than I could:
http://semiaccurate.com/2013/07/25/intel-keeps-up-the-unethi...
http://www.theverge.com/2013/1/9/3856050/intel-candid-explai...
If I can't use my laptop at the advertised clock speeds for more than 10-30 minutes continuously, then it's nothing but a scam, meant to fool consumers into thinking their chips are either a lot more powerful or a lot more efficient than they really are.
I think the benchmark manipulation by Samsung is in a different class; it configures the system in an impractical mode that actually isn't available to the end user at all.
In contrast, high-consumption turbo modes in laptop CPUs and GPUs really do benefit the end user, even if the benefit isn't everything you were hoping for.
ARM systems do the same tricks for the same reasons. Here an ARM CPU is throttled back to keep total system consumption below a target TDP, meaning it's impossible to achieve the rated CPU+GPU performance simultaneously. They are "configuring them to throttle back in order to meet the advertised thermal figures."
http://www.anandtech.com/show/6536/arm-vs-x86-the-real-showd...
That just strikes me as good engineering. If the clocks were given simplistic hard upper limits to achieve the same ends (4W TDP limit in a system otherwise capable of consuming 8W), the result would be a less flexible, less powerful system. A waste, really.
There is a real disparity between old-school absolute-maximum-TDP and realistic modern ultra-portable battery-powered use scenarios. Maybe Intel's SDP isn't the right answer, but it doesn't strike me as purely shady marketing malarkey either.
What is the issue here?
I mean really, even if you thought Android was a better solution why would you buy Samsung?
Whilst 'smart' behaviour might enable the phone to switch on some such functionality when the SoC/CPU is put under heavy load (by something such as a game), the effect of such automated switching is unlikely to have anything like the effect of a hardcoded force-to-high-power-mode as shown here.
It just guarantees that it will happen in that case. Unless you have some reason to believe that this mode is unattainable under any real world circumstance, I think the original point still has some credence.
I would guess that the high-performance mode drains the battery so quickly, or creates so much heat, that it's impractical for general use.
Based on the fact that they are Samsung. That they aren't transparent about what they are doing. That there is no evidence that these optimization kick in in other contexts.
They are doing something to fuck around with benchmark numbers, the onus is on them to explain it and be upfront about it.
The issue is that these 'optimizations' don't actually kick-in for non-benchmarking apps.
It also has a battery life extender for apps that measure battery life [2]
Don't forget the wifi range extender that boosts up the wifi decibel [3]
They also say their phones can use 16TB memory cards [4]
And don't forget that their phone is the prettiest [5]
1-4: I'm just making things up. 5: When compared to a naked mole rat.
So next time Samsung says anything about anything. Just remember that they are willing to stretch the truth more than the average company.