As if exploding batteries and chargers that burn down apartments weren't enough to tarnish the Samsung Galaxy S4's reputation, now we find out that it appears Samsung faked benchmark results for the device through some clever 'optimization' (code for cheating). So if you take benchmark tests as gospel and thought you were getting what you saw in those tests, you may have been a little hoodwinked.
A very clever poster in the Beyond3D forums by the name of Andrei F. was doing some GPU overclock and voltage control tests on the S4 and made a surprising discovery: it seems Samsung 'optimized' their GPU and CPU settings to run at higher speeds, specifically when benchmark tests were run. This means the numbers you see on benchmark tests are higher than you can get yourself.
In fact, when running GPU tests, Andrei found that the 533 MHz GPU clock that had been widely reported as the S4 benchmark, was only able to hit those heights when a specific benchmark test was launched to assess it. Any other time, I repeat, any other time, the GPU clock never ticked beyond 480 MHz – that's a 10% overstatement of performance. What this means is the S4 GPU only ever runs at 480 MHz, because since when do we consider running tests on a phone to be the same as using it?
What's worse is that he continued his testing and found basically the same thing with the CPU clock, but with a twist. Whereas with the GPU the higher frequency was only available to certain benchmark apps, with the CPU clock the 'optimization' was worked so whenever one of these benchmark tests was run, that the S4 would automatically pump up the power so it would appear the phone ran at a much higher processing level than it normally would. Kind of like puffing out your chest when a pretty girl walks by.
Basically, as soon as the benchmark test was launched, the S4 powers up the more powerful four of it's quad-core processors (in the Exynos 5 Octa device, the ARM Cortex A15 cores) and keeps them idling at nothing lower than 1.2 GHz, even if the device is just sitting there, doing nothing. Of course, when subjected to actual benchmarking, the device would then beef up further, and the overall average result would be tipped higher. Outside these benchmarks the results are dramatically lower, in many cases, a quarter of the speeds.
The same is true for Qualcomm S4's equipped with Snapdragon 600 cores. As AnandTech's Brian Klug reports: ''In these select benchmarks, the CPU is set to the maximum CPU frequency available at app launch and stays there for the duration, all cores are plugged in as well, regardless of load, as soon as the application starts.'' Again, Samsung is faking the performance of the S4 to make it post higher numbers than it normally would.
Note that Klug mentioned ''select benchmark tests.'' What this means is that for some strange reason, Samsung only 'optimized' the S4 for certain benchmark tests and not others. This loophole helped uncover this performance enhancing treachery. By only cheating with some benchmarking tests they left the door open to compare results against all benchmarks to identify the discrepancy. Rather sloppy work Samsung. If you're going to cheat, at least do it properly.
If Samsung had been more consistent with their cheating this may never have been discovered. Indeed it took this long for anyone to notice, and now only because of the 'select' choices of Samsung. Andrei F. happened to be delving into this area and, due to another oversight by Samsung, noticed that the GPU clock never hit 533 MHz. This oversight is more of a gimme from Samsung, however, whereby S4 owners can monitor the clock speeds of their device though an ADB command.
This provided the loose thread that before long unraveled the whole shebang. Why Samsung would cheat on benchmark tests is obvious - in some circles benchmark results are heavily relied upon to dictate what devices have better performance beyond simple hardware specs, and these results absolutely dictate purchasing behavior and tech blogging commentary.
But the strange part is that for the CPU at least, the performance is actually there – Samsung could only cheat because the S4 actually has the power available, they simply ensured the CPU performed at the highest rate when someone was looking. For the GPU however, only the benchmark tests can access the fabled 533 MHz, the rest of the time it is stuck at 480 MHz. This is the real issue at stake here: overbuild and under-perform.
As Brian Klug reported: ''Note that the CPU behavior is different from what we saw on the GPU side however. These CPU frequencies are available for all apps to use, they are simply forced to maximum (and in the case of Snapdragon, all cores are plugged in) in the case of these benchmarks. The 532MHz max GPU frequency on the other hand is only available to these specific benchmarks.''
While a lot of this may not mean much to the average consumer, it does mean that we are being lied to. This essentially constitutes cheating at the mobile device Olympics, as the benchmarks tests set by various smartphones go on to be used as all manner of advertising and justification for what's worth your hard earned dollars. Fancy that, false advertising and companies lying to us to get our money!
So what does this mean? Not much, the S4 seems to be doing a good enough job on its own of destroying its reputation, this just adds a little more mud to its already debatable reputation. And most consumers won't care about a few numerical discrepancies in performance tests. The good news is that this kind of thing will be checked for more closely by testers and bloggers. So Samsung should take note that they will be caught and just keep focusing on producing solid handsets with legitimate good performance, rather than investing valuable time and money in the smoke and mirrors department.
[Update 4:15 Berlin time 31.07.2013]
No sooner had the first stone been cast than Samsung replied on the ''faking the benchmark'' story with the following statement:
Under ordinary conditions, the Galaxy S4 has been designed to allow a maximum GPU frequency of 533MHz. However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode.
Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player and certain benchmarking apps, which also demand substantial performance.
The maximum GPU frequencies for the Galaxy S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results.
Draw your own conclusions, but we can't help but notice their new claims mention nothing about the specific code that AnandTech uncovered that specifically states 'benchmark booster'...