How Benchmarks Work And When You Should Pay Attention To Them

When you’re buying a new gadget, you want the fastest, most powerful one you can get for your money. Benchmarks can help you figure out what devices have the most power, but sometimes they’re just a distraction.

Photo by Michael Saechang.

What Does Benchmarking Really Do?

While we toss around words like “fast” and “powerful” to describe our gadgets, actually measuring their performance is a lot more complicated than it sounds. For example, you could measure the clock speed in a CPU to see how fast it is, but if you get a CPU with multiple cores it can get more done even if each core is slower. Graphics cards crank out huge amounts of processing power by chaining together thousands of relatively slow cores.

Benchmarks provide a way to measure real-world performance without getting bogged down in comparing specs. You don’t need to know how many cores or how much RAM a graphics card has if it renders Mass Effect: Andromeda at the highest frame rate in the most crisp detail. There are a lot of different kinds of benchmark tests, but here are some of the most common:

• Calculation tests measure a CPU’s processor power. At a basic level, processors are designed to do maths very quickly. Suites like Passmark use a variety of maths-based tests to see how much a processor can handle. For example, they might see how much data a CPU can compress in a given time frame, or how fast it can search for prime numbers. The more it can get through, the more powerful the processor is.
• Hard drive tests measure how fast you can read and write data. It doesn’t matter how fast your CPU is if your hard drive takes forever to find the data you need to process. Passmark tests how fast you can read, write and seek data on a hard drive to rank different hard drive models. The faster a drive can read or write data, the less it will bottleneck the rest of your system.
• 2D and 3D tests measure a graphics card’s performance. Suites like 3DMark and Heaven Benchmark test graphics cards and GPUs by seeing how fast they can display both 2D and 3D graphics. The more frames they can render per second, and the more detail they can render per frame, the better the score they get.
• Smartphone benchmarks measure each component in your phone. Your phone is a complicated mesh of components all specifically engineered to work together, which means isolating a single component is often harder than it seems. Suites like Antutu measure everything from CPU speed to how fast the interface is rendered, and even how fast you can read/write to your SD card or internal storage to give phones an overall performance rank.

Some benchmark tests use real-world situations to measure performance — for example, by playing Crysis — while others use more generic tests. In principle, they’re all designed to show how powerful a device is, but benchmarking also has its limits. For example, Samsung phones might score high on benchmark tests, but if Samsung fills up your phone with crapware, you could still end up with a slower phone than if you bought a lower-scoring, bloat-free Pixel phone. So, when do benchmarks really matter?

[referenced url=”https://www.lifehacker.com.au/2015/05/crapware-is-a-horrible-problem-and-its-all-our-fault/” thumb=”https://i.kinja-img.com/gawker-media/image/upload/t_ku-large/1260748651283262096.jpg” title=”Crapware Is A Horrible Problem And It’s All Our Fault” excerpt=”I love free apps. Who doesn’t love getting something for nothing? There’s just one problem. on the other side of that download link, every developer has to choose whether to charge money for their app, or offer it for free and find some other way to make money. And when we refuse to pay, we make that decision for them. We’ve created a demand for bundled crapware.”]

When You Should Use Benchmarks to Make Buying Decisions

Benchmarks are most useful when you’re measuring something specific like graphics card performance. They can also be handy if the type of device you’re looking at has so many specs and variables that it makes direct comparisons impossible. Again, like graphics cards. The following are situations where benchmarks can be the most helpful.

To Find the Best Graphics Card For Your PC Gaming Needs

If you’re building a gaming PC, you only need your graphics card to do one thing really well: Make your games look good. Companies like NVIDIA and AMD have so many different models and features for their graphics cards that finding the “best” one becomes a confusing mess. Benchmarks can be a huge help here.

Sites like Passmark and Futuremark keep a running list of graphics cards graded by their benchmarking suites. They also show the lowest price for each card from various online retailers. Futuremark also shows popularity scores for each card, so you can see what other users are buying. If you’re looking to upgrade, find your card on the list, then scroll up to find something with a higher score that’s within your price range. That will be the surest bet that you’re getting a better experience.

To Get a CPU That Can Handle Processor-Intensive Projects Like Video Editing

For the casual user, just about any CPU off the shelf will be enough to handle basic tasks like web browsing or checking email. However, if you do heavy tasks like editing video, graphic design or 3D modelling, you might want something more powerful. In that case, benchmarks can be helpful.

Passmark divides its rankings into several charts to help you narrow down which type of CPU you’ll need. The high end chart ranks CPUs designed for high performance gaming and editing rigs, as well as server-grade processors. For more moderate needs, Passmark’s high mid range list compares more traditional desktop CPUs if you just want to give your regular gaming PC a boost without spending a ton of money.

To Find Faster SD Cards For Your Camera, Phone or Raspberry Pi 

Slow SD cards can drag down the performance of anything you put them in. Whether they’re making your camera take forever to save pictures or slowing down the apps on your phone, they can become a painful bottleneck. SD cards are given class ratings that correspond to their minimum write speeds, but even within certain classes, some SD cards are much faster than others.

SD card benchmarks are a little harder to come by, but sites like Pi Dramble and TechFunology offer comparisons of some of the most popular cards. For average users, you can simply focus on getting a higher class SD card (Class 10 is the fastest, with a minimum write speed of 10MB/s), but if you need the absolute fastest card for your Raspberry Pi project or your professional photography camera, check out the sites above to find out which ones are the fastest.

When You Can Safely Ignore Benchmark Tests

For devices like smartphones that do a wide variety of tasks, generalised benchmarks can give you a vague idea of how powerful they are. However, your real-life usage is likely to be so different from the testing conditions that it renders the benchmark almost meaningless. Here are some situations in which you might be better off ignoring the tests.

It’s OK to Look At Smartphone Benchmarks, But Don’t Buy Based On Them

Antutu keeps a running list of smartphone benchmarks, ranking them according to their performance. This is good information to have, but it should hardly be the only factor you consider. A phone that gets a good benchmark score might get poor battery life or have a crappier camera.

More importantly, the benchmark score only matters if you’re planning to push your phone to its limits. If you use your phone for checking Gmail and reading the news, a crazy powerful processor won’t do you much good. It might be better for playing mobile games, but that depends highly on how well game developers utilise that power. If you want to future-proof your phone, it might be a good idea to pick something in the top 20 or so, but don’t sacrifice other features just to get a higher benchmark score.

It’s also worth pointing out that the highest-ranking Android phone on Antutu’s benchmark list is the OnePlus3T, which was accused of artificially inflating its benchmark score. While the mechanism they used to do this is complicated and it’s not quite cheating, it also means that the score isn’t necessarily reflective of everyday usage. This isn’t the first time that a manufacturer has been accused of trying to tweak their system to get favourable benchmarks. This is yet another reason that you should take phone benchmarks with a grain of salt and not the foundation of your buying decision.

Skip the Hard Drive Benchmarks and Just Buy An SSD

Sites like Passmark and UserBenchmark offer comprehensive lists ranking both old school platter hard drives, as well as solid-state drives. Unless you’re building a server for a business, however, you can ignore them. If you’re using a platter drive, the best speed boost you can possibly get will be to get an SSD. If you already have an SSD, you’re better off buying based on how much data they can hold or how reliable a particular model is.

Some old school platter-based drives are faster than others (for example, 10,000 rpm drives will be better than 7200 rpm ones), but SSDs don’t use platters at all. This makes them much faster. Almost twice as fast as regular HDDs, in fact. If you have an older style hard drive, don’t bother with benchmarks. Just get an SSD. If your computer is still slow, you’ll probably see a bigger benefit by upgrading your other components.

Depending on your needs, you might care more about benchmarks than the average person and that’s OK. If you’re building a high-performance NAS or buying a phone to write apps on, you might want to pay more attention to benchmarks. However, in general, benchmarks are a useful tool that you shouldn’t always treat as gospel.