The Android Benchmarks You Can Actually Trust

We all know companies try to present benchmarks that show their gadgets in the best light possible, but you can only fudge the numbers so much before they graduate from mere flattery to flat-out lies. Anandtech decided to take a look at Android benchmark claims and found out most were way off.

It seems the only companies who don’t cheat are Google/Motorola and NVIDIA. (Apparently Apple doesn’t either, but it obviously does not make Android devices.) Everyone else? They inflate half or more of their test scores. That includes Samsung, HTC, LG and Asus. They all run CPU optimisations to essentially game the benchmarks and make performance look better than it really is. Strangely, as Anandtech explains, this doesn’t amount to much:

The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 – 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can’t stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we’ve seen how this story ends. We’re very much in the mid-1990s PC era in terms of mobile benchmarks. What follows next are application-based tests and suites. Then comes the fun part of course. Intel, Qualcomm and Samsung are all involved in their own benchmarking efforts, many of which will come to light over the coming years. The problem will then quickly shift from gaming simple micro benchmarks to which “real world” tests are unfairly optimised which architectures. This should all sound very familiar.

Of course, even without the gaming, benchmarks don’t tell you much about actual use of a device, so you shouldn’t put too much stock in them in the first place (especially with smartphones). You can learn a lot more about how performance actually affects your day-to-day activity by trying out things you regularly do on a device rather than hoping the numbers make sense. For now, the data will lie and you should only trust your experience.

They’re (Almost) All Dirty: The State of Cheating in Android Benchmarks [Anandtech via MacRumors]