Huawei caught cheating performance test for new phones
techcrunch.com
techcrunch.com
Sadly the P20 pro has an incredible camera and would stand on its own without forgery.
All I really care about is that they unlock their bootloader. The marketing crap is always half truths anyway. Retina, you mean 4k?
I cant find this, do you have a link? I'm quite interested in what they do.
https://fstoppers.com/education/new-iphone-fashion-shoot-bik...
https://fstoppers.com/commercial/iphone-fashion-shoot-lee-mo...
For big money advertising shots, I'm sure Apple uses gimbals/tripods. I haven't seen any proof they attach giant Sony/Nikon lenses though.
I would be inclined to be more aggressive than that. Leave them listed, but add support in the site or other publication system to display "deliberately broke the rules of the test" as a negative, perhaps displaying in a negative looking colour and in any graphs show as a if they scored, say, 20% worse then the worst other score.
If presenting some arithmetically derived overall rating of benchmarks and other properties, balloon this further. Then if the rest of the phone is demonstrably brilliant they still have a hope of not looking terrible, but the cheating hurts the result otherwise.
Unfortunately in these litigious times such an inclination would probably cause me some financial trouble, so it is probably for the best that I don't run a device comparison site! This may be why simply delisting was the chosen way of handling the situation here.
Not sure if it was ever reverted eventually but performance test might not matter to HN users but it is something that typical consumers do look at when making purchasing decisions, based on our research.
That a lie increases sales is at the top of the list of bad reasons to lie to customers (and they are all bad reasons).
I think that in the society we've built it's actually one of the best reasons to lie to customers. Advertising is pretty much built on that.
It's nearly impossible to be "ethical" in large OEM companies because of the extreme competition and complexity of the organization. Most of these large OEMs are so political that even the smallest changes would require way too much effort.
But Apple brags endlessly about their A11 "Bionic" chip and similar[0].
Arguing that Apple values experience over specs might have been a valid argument back with Steven Jobs, but Apple's presentations spend significant amounts of time talking about hardware specs these days.
I don't specifically mind that Apple does, but people need to stop pretending like Apple hasn't changed as a company with the shift in leadership.
What's Option B? Advertise based on what they're not good at? "Snapple: Made from the Fifth or Sixth Best Stuff on Earth"
People who are really serious about software should make their own hardware
(That's what made me pay Google directly for the first time; extra storage for backing up photos.)
For the hobbyist or pro photographer backing up their full frame RAWs, sure, you want a more professional (read: paid) backup solution but that doesn't seem unfair to me. For the 95% of users who want to free up space taken up by vacation snaps and selfies, compressing photos over 16MP is often acceptable when it doesn't cost extra.
full picture https://images.huffingtonpost.com/2016-01-01-1451631423-7665...
cropped detail https://images.huffingtonpost.com/2016-01-01-1451631515-2060...
Also, I believe Photos storage is unlimited even in Original quality if you own a Pixel phone, which may or not may be the case, but still.
I'm wary of services that auto-compress images, because a) I want my bits to stay the way they were originally captured, and b) every now and then I have to print a photo I make, and then suddenly all those details matter for the print quality.
Bought an iPhone 6S second hand for $320. Still getting updates now, and looks like I will for the foreseeable future.
The core feature iPhones are missing is USB Type C- I won't upgrade to a new iPhone if I cant use the same headphones on my laptop and phone (either 3.5mm or TypeC)
To fix this, there should be different runs for these benchmarks. A max-clocks run with the phone set in an actively cooled cabin to see what the hardware can theoretically do at its very best, and another run in non-actively cooled room temperature with the test running for about an hour to rule out any benefits from temporary boosts or ignoring thermal limits to get an idea of continuously supported performance.
Even those results wouldn't really tell much to the end-user. I always suggest my friends and relatives to buy a phone with excessively large memory and storage space because lack of memory and storage really is what turns phones slow after a few rounds of application and system updates.
The performance edge on mobile is thinning out pretty much what happened with PCs. Ditto for memory and storage. Early on, vendors competed on who has the highest MHz cpus but somewhere between 1-2GHz the performance got high enough in nearly any case. 95% of people could buy about just any PC or laptop and it would be "fast enough". There's gigabytes of memory in even the sloppiest laptop these days, and enough SSD to make things fly. The same will happen with phones which makes it impossible to buy a phone that is too slow. At that point components with lower performance and less capacity will become more expensive due to the lack of volumes, that no vendor will bother any longer.
[1] https://play.google.com/store/apps/details?id=com.glbenchmar...
It probably won't come as a surprise to many of you, but there's a major conflict of interest when it comes to benchmark companies. They have "benchmark development programs" which companies pay tens or hundreds of thousands of dollars to join, and in turn the member companies get early access to beta versions of the benchmarks, and get to propose optimizations to the benchmarks.
Now the benchmark companies insist that they make impartial decisions when it comes to what to include or improve in the benchmarks, but you have to wonder just how much influence these member companies have over the benchmark companies, especially when it comes to new features that are in one but not all devices, or features that are not often used by developers but have different performance on different devices, what the decision process is to include tests for those features.
>In other words, the phones aren’t so smart after all.
If I may, also the fact that benchmarks are normally/usually "recognizable" through some "label" doesn't seem "smart" to me.
I don't really know what's their roadmap for Western markets considering nobody wants to let off their Google world. Even though the cynics among us will placate it as the Party bankrolling it for spying.
But I don't think I'll get another Huawei next time, because the company itself is really weird.
I spoke a fair bit about performances with a Googler in charger of it. Apparently he is appalled that OEMs have been optimizing their phones in order to score high on these benchmarks.
They don't reflect real use at all and are not what should be optimized against.
In the end, he documented the work he did on the various terminals he worked on so OEMs could do what they should have been doing for years..
Also all the outrage when people discovered that a Radeon 9600 was just a Radeon 9800 running in crippled mode, and that you could bridge the tracks and re-enable it in 9800 mode. (Or something similar, anyway, that was a long time ago... :P )
Also on a completely different note, the 9700 Pro which was the flagship card of that generation, cost about $300. Times have changed.
Source code for their AI engine:
if (app == benchmark) { increasePerformance(); }
They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps.
We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also lined up with our normal workloads so we had a decent idea of where we expected the whole system to land.
https://www.hardocp.com/article/2001/10/23/optimizing_or_che...
TLDR: ATI did the same thing with Quake 3 Arena which was commonly used as a benchmark game for testing new video cards of that era and which many game news sites were using.
It did go too far, even to the extreme of Nvidia hardcoding in their GPU driver the exact camera path that 3dmark took ( https://www.extremetech.com/computing/54154-driver-irregular... ), which is more like what the mobile space is grappling with now. That was solved through a combination of scolding and by just changing the benchmarks to be what people actually do. You don't see 3dmark show up in GPU reviews much, you instead see performance results in a handful of major games. If mobile follows suit and shifts to doing things like benchmarking the average/90th/99th percentile of a game of Fortnite or something then these cheating practices become less feasible as a whole.
Yep. That was the problem at he time. There were no shaders/etc so you’d just taking a demanding game and see how good it ran. No one optimized specific games in their drivers.
So what ATI did made it appear the GPU was stronger on average that it was.
If it was disclosed it would have been a feature and might have been welcomed. But it was secret.
I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates?
I forgot about the nVidia “micro optimization for a specific benchmark” one. No good excuse there because no one actually plays it.
Well, not really. It really did run Quake 3 at the speed the benchmarks were reporting. The problem was it got that speed at the expense of a drop in image quality, and a somewhat substantial one at that.
But if you launched the game and just played it you did get the exact same performance that the benchmarks were reporting.
> I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates?
Desktop GPU optimizations or mobile ones? Desktop GPU optimizations these days are not considered cheating because of 2 key points: 1) it improves the actual game experience (this is why reviews focus on game performance now) and 2) it doesn't impact image quality (periodically you'll see image quality comparisons to ensure both sides are still being honest)
The mobile things that Huawei are doing though are cheating because it does not translate into improvements in actual user experience or gameplay. It's tuning the thermal & power consumption to hit the throttling point at the exact duration of the benchmark. This doesn't translate in any way to any real-world improvements. Thus, cheating.
GPU drivers are still loaded with such replacements, either to work around bugs or to optimize games where a developer partnership was impossible or otherwise rejected. I think that with the shift to lower-level GPU APIs this practice is beginning to fade, but it's basically been par for the course since programmable shaders became a thing.
However, I agree that shader replacement is great for the consumer especially in situations where developers have agreed to vendor lock-in/specialization agreements. It's only questionable to me in the sense of benchmarking applications.
Except Apple.
However given that Apple isn't the only one who licenses their designs I'd put a large sum of money on them doing the same thing.
Issues like the Intel i9 performance going significantly downward with consistent usage are often missed by benchmarking apps: https://arstechnica.com/gadgets/2018/07/we-tested-throttling...
My 3 year old Oneplus X is more responsive than my iPhone 8 (work phone).
We had one device I was helping bring up where you could run the GPU full-tilt for ~60s. After that the device would reboot and fail to come up until it had cooled enough. Was a fun process characterizing that device and our workloads in 60s bursts. Couple weeks later they had it tuned in where you could run it 24/7 as the thermal limits kicked in pretty aggressively.
Battery life is not the only concern; a device that sits in your hand or your pocket needs to carefully manage heat.
Power management and thermal budgets are a very real thing. The times where all device’s components would run at full speed are (thankfully) over for quite some time.
20 years ago PCs had multiple incredibly loud constantly running fans, that I think hardly anyone would put up with anymore (and would obviously not work for phones anyway). Today’s phones are extremely more powerful than those PCs in many ways, and yet consume a fraction of energy per time unit. Does not work without complex power management.
The industry didn't focus much on quietness, that's true.
(I used to assemble and sell custom-built PCs around 1996 to 2000.)
Even with computers that could be run all passively at that time, there is just no comparison to today's smartphones, which sit in our pockets and are orders of magnitude more capable along almost every axis.
Quack.exe was for Quake 3, Before shaders even existed. At that time drivers didn’t have special paths for certain games, which is what made it a scandal.
If ATI had said they were doing it, perhaps it would have been fine. But they hid it and looked like they were doing something underhanded (true or not).
Besides, if you want to install custom roms just get a phone that is supported officially (e.g. Pixel). I don't care about custom rom scene so it doesn't bother me that the bootlocker is locked. Though it was definitely a nice to have while it still existed.