Smartphone benchmarking – what do these numbers mean?


fonearena_001

Off late, we have been receiving a lot of requests from you folks asking us to detail about the smartphone benchmarking. We heard you and today, we try to explain what these numbers really mean to you.

First up what is smartphone benchmarking? Well, it’s quite impossible for anyone of us to numerically and even scientifically evaluate the performance of a mobile phone by just using it. Of course, we can gauge how smooth is its performance, how cool is the animations, how quickly applications open and close among other similar aspects.

However, it’s almost impossible to evaluate the internal performance of a mobile device and you never know how well the hardware performs. This is where the smartphone benchmarking applications come in handy and to an extent they solve the issue.

You might have noticed we at FoneArena run multiple benchmarking tests for every smartphone that makes its way to our labs. The reason behind running different benchmarking test is the fact that some applications are designed to evaluate only a particular segment of the phone.

Today, we would like to give you an insight of what we do and if you ever thought reviewing a smartphone is a simple task – we don’t think you will have the same perspective after going through the entire story.

Benchmarking each smartphone literally takes hours. We basically begin the test, wait till it finishes evaluating and record the result. It’s just not done yet. You start all over again and perform the same operation a couple of times. Mind you this is only for one benchmarking test.

The reason behind performing the test in multiple iterations is because the result varies each time. So, why does that happen? The working of the smartphone CPU is quite complex. At times it might use just two cores to do a job, while the same job might require four cores if there are multiple applications in the background. So, the evaluated performance is not even all the time. This is why we run multiple iterations and later average them out.

It’s always a wise thing to take these benchmarking numbers with a grain of salt and there is multitude of reasons why you should do that. In any field that involves electronics there is tolerance. So, the performance is not always the same. Having said that – the consistence of all the electronic components have greatly improved.

Secondly, an impressive score doesn’t necessarily mean the device will provide top notch performance all through the day. For example, your smartphone might show some traces of lag and stuttering by the end the day due to excessive usage of the device.

Now, you must be wondering what does the benchmarking score actually mean. Well, get all ears. These numbers are just a reflection of the horsepower the device has to muscle through the basic every day task. In a nutshell, the real world performance and the benchmarking numbers don’t always go hand in hand.

Another biggest reason not to entirely put your money based on the benchmarking numbers is the accuracy of the results itself. With new technology being developed, manufacturers are coming up with new chipsets with latest developments. So, it’s obviously hard for the developers of benchmarking applications to support various processor technologies. For example, the compiler used for evaluating ARM-based silicon’s is not the same for gauging the performance of an Intel based chipset.

The smartphone industry went through a scary face about a year and a half ago. The over dependency on these benchmarking numbers meant that OEMs started to optimize their software to assist their devices performance in benchmarks instead on focusing on optimizing the overall performance.

The problem wasn’t specific to any manufacturer and it happened with custom ROMs as well. There was a huge boost in the result of the device with a custom ROM and kernel flashed, while the stock ROM on the same device failed to impress with its result. The enhancement in the benchmarking test meant literally meant nothing as there were no improvements in the everyday performance.

These is why benchmarking doesn’t deserve our full faith. At the same time, marking all the benchmarks as useless is not a great approach. While these numbers may not be accurate and may not go well with the real world performance, they are one of the best ways to quantitatively put two devices together.

The best way to evaluate a mobile device is to balance the real world performance and the benchmarking numbers of the phone. The benchmarking scores will never be able to tell you if the device lags, while the real world performance can never show you wants happening inside. If you can analyse both the things together, you will pretty much know what to expect from the mobile device when you lay your hands on it.

We are going a step further and listing out some of the most popular benchmarking applications –

AnTuTu Benchmarking – It is one of the most comprehensive Android benchmarking applications available out there. The tests conducted by the app includes memory performance, CPU integer performance, CPU floating point performance, 2D / 3D graphics performance, SD card read-write speed and Database IO performance testing.

fonearena_002

Geekbench 3 Benchmarking – It is a cross platform processor benchmarking application with a scoring system that separates the single core and multi core performance of a CPU. It also has workloads that simulate real-word scenarios.

fonearena_003

Quadrant Benchmarking – It is a benchmarking application that runs a series of processor intensive tests to determine the performance of the device. The app is capable of measuring CPU, memory, I/O and 3D graphics performance.

fonearena_004

Basemark Benchmarking – It measures next generation game graphics performance. The application utilizes new OpenGL features such as 3D textures, instanced rendering, transform feedback and occlusion queries. It also features contents that reflect real-life use case scenarios.

fonearena_005

GFX Bench – It is another cross platform and cross API 3D graphics benchmarking application that measures graphics performance, long term performance stability, render quality and power consumption of a device.

fonearena_006

Vellamo Benchmarking – It is a web browser comparison benchmarking tool that evaluates browser performance on mobile devices. The app provides analysis of browser performance and its stability, including networking, JavaScript, rendering and user experience.


Author: Sunil Jamboor Nagesh

He has been reporting tech and reviewing gadgets for quite some time now. SJN in the past has been a part of couple of globally reputed tech publications like GSMArena and AndriodAuthority. He bleeds tech and is always seen with a review unit in hand. Feel free to reach him on twitter - @SunilJamboor or on Google+.