Cheating. The word reminds me of fielding too many players in a football game or using answer sheets in school. But even in the world of technology, there is a lot of cheating going on by companies. This time Huawei got caught... so what?
Anandtech's experts are considered smart and incorruptible when it comes to chips, processors and laboratory tests. Now, the tech heads have discovered that Huawei is boosting the performance of both its own and the Honor smartphones in benchmark tests - making the smartphone run faster than it does in everyday life if no benchmark app is recognized. Anandtech confronted Huawei directly with the findings, and the manufacturer does not deny the behavior. Quite the contrary, in fact.
Huawei provides an explanation, or let's call it an excuse, that everyone does it anyway, especially the competition in China. You can't score points with customers with honesty in this area. That may be true, but you shouldn't get caught. Take a look at what happened to Volkswagen after the exhaust emissions scandal in the automotive industry, for example. But at the end of the day, Huawei is right: These tricks are nothing new in the mobile industry, and even if no one has been conspicuous about them recently, it is an open secret that everyone cheats the benchmark tests.
Everybody cheats and, in the end, everyone loses
This whole saga reminds me a bit of professional sport. When so many athletes dope, as in the Tour de France for example, it ruins the reputation of the sport, but the competition between them is no less fair than it already is due to the varying resources or financial backing the teams have.
Just like in the Tour de France - and this is really just one example, I could have mentioned many other sports and events - where I find it a pity that the competition suffers from fraud and the sport comes into disrepute, I just don't really care about smartphone benchmarks anymore. There is no well-known manufacturer that doesn't try to present a good benchmark score. Huawei is anything but alone in this.
Huawei says it always prioritizes the user experience
Huawei reached out to us to provide the following official statement on the story:
"Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.
"When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.
"In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to “Performance Mode” and delivers optimum performance. Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to.
"Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience."
Without standards and control, tests are worth only half as much
Yes, of course, the tests give us an indication of how powerful a platform is. And yes, they allow for a certain level of comparability. But as long as the tests are not standardized and controlled in any form, they provide nothing more than an educated guess and are too easy to manipulate. They are important as just one of many indicators in a performance test but are by no means decisive. The scores should only to be considered rough guides. A few hundred points either way does not make much difference.
No matter how good the benchmark scores are, if a smartphone shows weaknesses in everyday life, customers and testers will definitely notice that. That's real life, and no matter how fabulous your stats are, it won't help you. Let them all cheat, we'll still find out whether the smartphones are fast or lame when we use them in real life. You can always rely on that.
What do you think about the relevance of benchmarks today? Let us know in the comments.