Processor Doping: Do Smartphone Benchmarks Really Matter?
In the early days of Android, disparity between processors, clock speeds, and the manufacturers of the ARM-based CPUs meant that there was no objective way to compare a phone’s performance. Benchmarking tools helped to solve that by normalizing the curve and comparing processor and graphics performance, applying applicable tasks that can be repeated regardless of make or brand of processors. However, given the advances in the smartphone and mobile technology market, benchmarking stats may not be a reliable indicator of how a phone performs in the real world, and in the end may not be a good tool to base your next smartphone purchasing decision on.
Samsung in the Heat
With the international variant of the Galaxy S4 based on Samsung’s Exynos 5 octa, a big.LITTLE ARM architecture with 8 cores but where the chip uses only 4 high performance or energy saving cores at any given time, receiving accolades for the device’s performance, Samsung found itself recently in the hot seat as reports flooded in that the company may have artificially boosted performance. The artificial boost gave Samsung’s flagship smartphone and its processor a bump in the benchmarking scores.
It should also be noted that the Galaxy S4 sold in the U.S. and U.K. markets rely instead on Qualcomm’s Snapdragon 600 processor, a decision that Samsung’s executives say was made due to supply constraints behind the Exynos 5 octa at the time of the phone’s debut. As such, U.S. variants aren’t affected by the benchmark debacle that we know of at this time.
In response to the AnandTech discovery of Samsung doping its Exynos chip, the company has denied those allegations.
“The maximum GPU [graphical processing unit] frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results,” Samsung said in a statement. “A maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.”
And according to All Things D, Samsung indirectly acknowledged that benchmarking tools got a boost as a result of this. Samsung boosts the graphics peformance of select first-party apps to allow them better performance, but it also does that with “certain benchmarking apps” as well.
Real World Performance
And while there may not be any real harm done with Samsung doping its benchmark scores a little–the chip still is a fast performer and is a high-end contender in the mobile processing space, benchmark scores don’t consider real-world usage. In a sterile environment of playing back videos, checking for frame rates and graphics when playing the most intensive games, and analyzing text and graphic rendering, benchmarks tell a different story than the varied use of our phones in the real world environment.
Many consumers will multitask and juggle between web browsing and phone calls, text message their friends and reply to emails, watch YouTube and Netflix videos, launch a number of apps where a majority of them won’t be high performance games, and they may launch a news app, search the web, and check sports scores.
Consider Samsung’s push for the Multi Window view to allow users to simultaneously multitask by opening two apps at the same time side-by-side. In order to do that, and still offer a great user experience, Samsung must be confident with the technologies available in order to deliver this feature.
With normal use, many of today’s phones–even a lot of the inexpensive entry-level phones–can still deliver a fluid experience.
Software Code is Required to Optimize
Benchmarks that rely solely on the hardware isn’t telling a complete story anymore. Motorola’s Moto X smartphone is said to be using an ARM-based X8 processor that’s based largely on a dual-core Snapdragon S4 Pro, but with additional cores to provide intelligent, probabilistic computing. This requires software optimization to take advantage of everything that Motorola has to offer, and third-party developers will likely need to tap into the software tweaks Motorola has created.
The same is true for the larger OS as a whole. Apple says that in the post-PC revolution, tech specs don’t matter as it’s the about the marriage of software and hardware. With each successive OS iterations, Apple, like Google’s Android, optimizes the operating system to perform better and handle more workload despite having the same hardware as before. This leads to a more fluid experience and so benchmarks and hardware specs only tell one part of the story. The other part may be told by the OS, any manufacturer optimizations, how UI overlays like TouchWiz perform, and if there are APIs for third-party apps to take advantage of so that these apps would get the same ‘boost’ in performance like native apps would.
Benchmarks Don’t Matter
Sure, you can dope the system and as a result get some artificial boost in scores clocked by these benchmark tools, but at the end of the day if the phone doesn’t perform, you’ll end up with angry customers and a high return rate. This doesn’t seem to be the case at all with the Galaxy S4.
Despite some concerns with S Lag early on, the Galaxy S4 still is a performance machine. In reality, after a certain threshold, users may not even notice any marginal gains in performance.
Other Factors That Matter
In a smartphone experience, there are other factors as well that matter. A network’s connectivity speed may give a heightened sense of performance as well, especially since smartphones revolve around hyper-connectivity. A fast processor and slow network speed would still mean constant video buffering or long webpage loading times, which would give users the false impression that the phone may be slow. This applies both to WiFi networks as well as to mobile broadband 3G and 4G connectivity.
Phones That Prove Benchmarks Wrong
Despite its low $99 asking price–and that’s without a contract even–the Nokia Lumia 520 with its more modest specs and limited RAM shows that despite having a humble beginning, it could still be a champion. The device isn’t a speed demon, and launching apps do take a second or two longer than on a high-end Lumia 1020, but the Lumia 520 still handles 85 percent of my normal day-to-day task extremely well.
Compare that to a budget Android handset, and another reason why benchmarking doesn’t matter is that Windows Phone is able to handle more conservative hardware better. So even between OSes, hardware, for which benchmarks are based off, may not tell much about the experience. Windows Phone’s more conservative use of resources make it more fluid even on low-end devices than Android.
What You Should Consider When Purchasing Your Phone
In the U.S. phone users are locked into a two-year contract typically, unless they’re willing to pay full price for their phones or pay an early termination fee to exit out of that nuptial with the carrier. In any case, when shopping for a phone, you should consider your usage today while at the same time forecasting how your usage will change. This will allow users to have a phone that grows with them. Expandable storage and removable batteries could be examples of how the phone grows. But so is cloud storage, which relies more on faster connectivity speeds to the Internet than processor speed. If you’re a heavy gamer, you should opt for a fast GPU, which in all likelihood means you’ll end up with best processor on the market today. Casual gamers who play connected games may not care much about any of the high-end specs. The reality is that most mid-range phones are good enough for 80 percent of the population and at the end of the day, if you’re in the majority, Samsung’s headache today will all be too quickly forgotten as it really doesn’t matter.