Meanwhile, the run with enabled cheating spent a staggering 95% of readings in its maximum frequency state. In case it isn’t evident from the graph above: we polled the CPU frequency every 100ms, and in total, only 24.4% of readings returned the maximum frequency of 1.9Ghz when disabling cheating. Below you can see a frequency over time plot for the OnePlus 5’s little cluster when running GeekBench 4 from the Play Store, and the same configuration running a build of GeekBench 4 stripped of identifiers that is able to fool OnePlus’ cheating mechanism. We found that while running GeekBench 4 from the Play Store, the device scored over 6,700 in multi-core, while we never obtained a score of 6,500 once the device behaves as expected with our hidden build of GeekBench.
We were able to spoof the benchmark cheating and evade it with GeekBench 4, similarly to our testing in our last report. The difference in scores is just what you would expect, for the most part. What is completely unsurprising is the applications affected are the exact same ones as last time around, and OnePlus is clearly targeting the very same packages. Below is a list of benchmark applications affected: Scores certainly higher than those obtained by similar devices and Qualcomm’s own MSM8998 test device which we were lucky enough to benchmark.
All little cores are affected and kept at 1.9GHz, and it is through this cheat that OnePlus achieves some of the highest GeekBench 4 scores of a Snapdragon 835 to date – and likely the highest attainable given its no-compromise configuration with its specific configuration. While there are no governor switches when a user enters a benchmark (at least, we can’t seem to see that’s the case), the minimum frequency of the little cluster jumps to the maximum frequency as seen under performance governors. The OnePlus 5, on the other hand, is an entirely different beast - it resorts to the kind of obvious, calculated cheating mechanisms we saw in flagships in the early days of Android, an approach that is clearly intended to maximize scores in the most misleading fashion.
#Deleted from geekbench over cheating allegations full#
In short, cheating behavior was clear and demonstrable by both looking at score variance, and by monitoring CPU frequencies throughout the benchmark, which showed a frequency floor that – for the most part – allowed the device to consistently score closer to its full potential. This minimum frequency reduced the effective frequency range, which in turn reduced the number of step frequencies in benchmarks, this resulted in slightly lower variance and, as we showed, higher sustained performance as the higher minimum frequency could not be overridden by thermal throttling. Then, the ROM would alter the frequency in relation to an adjusted CPU load - our tools showed CPU load would drop to 0% regardless of obvious activity within the application, and the CPU would see a near-minimum frequency of 1.29GHz in the big cores and 0.98GHz in the little cores. Such application names were explicitly listed by their package IDs within the ROM in a manifest that specified the targets. So how does it work, and what’s the difference? Last time around, OnePlus introduced changes to the behavior of their ROM whenever it detected a benchmark application was opened. What is worse is that, this time around, the cheating mechanism is blatant and aimed at maximizing performance, unlike last time which did not increase scores by much on average, but did reduce variance and thermal throttling, as we found.
As a result, every OnePlus 5 review citing benchmark scores as an accolade of the phone’s success is misleading both writers and readers, and performance analyses based on synthetic benchmarks are invalidated. This is an inexcusable move, because it is ultimately an attempt to mislead not just customers, but taint the work of reviewers and journalists with misleading data that most are not able to vet or verify. Unfortunately, it is almost certain that every single review of the OnePlus 5 that contains a benchmark is using misleading results, as OnePlus provided reviewers a device that cheats on benchmarks. While no customers have a device in their hands (it just launched after all), we have learned about OnePlus’ new benchmark cheating mechanism through our review unit, which we received about ten days ago before the day the embargo breaks and reviewers are allowed to report on the device. Today, we sadly must follow-up on our accusations as the company has once more been inappropriately manipulating benchmark scores in the OnePlus 5. Earlier this year, we published a report that denounced OnePlus (and other companies) for their improper behavior in regards to benchmark manipulation on newer builds of OxygenOS.
0 kommentar(er)
