How Benchmarks Work And When You Should Pay Attention To Them

When you're buying a new gadget, you want the fastest, most powerful one you can get for your money. Benchmarks can help you figure out what devices have the most power, but sometimes they're just a distraction.

Photo by Michael Saechang.

What Does Benchmarking Really Do?

While we toss around words like "fast" and "powerful" to describe our gadgets, actually measuring their performance is a lot more complicated than it sounds. For example, you could measure the clock speed in a CPU to see how fast it is, but if you get a CPU with multiple cores it can get more done even if each core is slower. Graphics cards crank out huge amounts of processing power by chaining together thousands of relatively slow cores.

Benchmarks provide a way to measure real-world performance without getting bogged down in comparing specs. You don't need to know how many cores or how much RAM a graphics card has if it renders Mass Effect: Andromeda at the highest frame rate in the most crisp detail. There are a lot of different kinds of benchmark tests, but here are some of the most common:

  • Calculation tests measure a CPU's processor power. At a basic level, processors are designed to do maths very quickly. Suites like Passmark use a variety of maths-based tests to see how much a processor can handle. For example, they might see how much data a CPU can compress in a given time frame, or how fast it can search for prime numbers. The more it can get through, the more powerful the processor is.
  • Hard drive tests measure how fast you can read and write data. It doesn't matter how fast your CPU is if your hard drive takes forever to find the data you need to process. Passmark tests how fast you can read, write and seek data on a hard drive to rank different hard drive models. The faster a drive can read or write data, the less it will bottleneck the rest of your system.
  • 2D and 3D tests measure a graphics card's performance. Suites like 3DMark and Heaven Benchmark test graphics cards and GPUs by seeing how fast they can display both 2D and 3D graphics. The more frames they can render per second, and the more detail they can render per frame, the better the score they get.
  • Smartphone benchmarks measure each component in your phone. Your phone is a complicated mesh of components all specifically engineered to work together, which means isolating a single component is often harder than it seems. Suites like Antutu measure everything from CPU speed to how fast the interface is rendered, and even how fast you can read/write to your SD card or internal storage to give phones an overall performance rank.

Some benchmark tests use real-world situations to measure performance — for example, by playing Crysis — while others use more generic tests. In principle, they're all designed to show how powerful a device is, but benchmarking also has its limits. For example, Samsung phones might score high on benchmark tests, but if Samsung fills up your phone with crapware, you could still end up with a slower phone than if you bought a lower-scoring, bloat-free Pixel phone. So, when do benchmarks really matter?

Crapware Is A Horrible Problem And It's All Our Fault

I love free apps. Who doesn't love getting something for nothing? There's just one problem. on the other side of that download link, every developer has to choose whether to charge money for their app, or offer it for free and find some other way to make money. And when we refuse to pay, we make that decision for them. We've created a demand for bundled crapware.

Read more

When You Should Use Benchmarks to Make Buying Decisions

Photo by Jack-Benny Persson.

Benchmarks are most useful when you're measuring something specific like graphics card performance. They can also be handy if the type of device you're looking at has so many specs and variables that it makes direct comparisons impossible. Again, like graphics cards. The following are situations where benchmarks can be the most helpful.

To Find the Best Graphics Card For Your PC Gaming Needs

If you're building a gaming PC, you only need your graphics card to do one thing really well: Make your games look good. Companies like NVIDIA and AMD have so many different models and features for their graphics cards that finding the "best" one becomes a confusing mess. Benchmarks can be a huge help here.

Sites like Passmark and Futuremark keep a running list of graphics cards graded by their benchmarking suites. They also show the lowest price for each card from various online retailers. Futuremark also shows popularity scores for each card, so you can see what other users are buying. If you're looking to upgrade, find your card on the list, then scroll up to find something with a higher score that's within your price range. That will be the surest bet that you're getting a better experience.

To Get a CPU That Can Handle Processor-Intensive Projects Like Video Editing

For the casual user, just about any CPU off the shelf will be enough to handle basic tasks like web browsing or checking email. However, if you do heavy tasks like editing video, graphic design or 3D modelling, you might want something more powerful. In that case, benchmarks can be helpful.

Passmark divides its rankings into several charts to help you narrow down which type of CPU you'll need. The high end chart ranks CPUs designed for high performance gaming and editing rigs, as well as server-grade processors. For more moderate needs, Passmark's high mid range list compares more traditional desktop CPUs if you just want to give your regular gaming PC a boost without spending a ton of money.

To Find Faster SD Cards For Your Camera, Phone or Raspberry PiĀ 

Slow SD cards can drag down the performance of anything you put them in. Whether they're making your camera take forever to save pictures or slowing down the apps on your phone, they can become a painful bottleneck. SD cards are given class ratings that correspond to their minimum write speeds, but even within certain classes, some SD cards are much faster than others.

SD card benchmarks are a little harder to come by, but sites like Pi Dramble and TechFunology offer comparisons of some of the most popular cards. For average users, you can simply focus on getting a higher class SD card (Class 10 is the fastest, with a minimum write speed of 10MB/s), but if you need the absolute fastest card for your Raspberry Pi project or your professional photography camera, check out the sites above to find out which ones are the fastest.

When You Can Safely Ignore Benchmark Tests

Photo by Laineema.

For devices like smartphones that do a wide variety of tasks, generalised benchmarks can give you a vague idea of how powerful they are. However, your real-life usage is likely to be so different from the testing conditions that it renders the benchmark almost meaningless. Here are some situations in which you might be better off ignoring the tests.

It's OK to Look At Smartphone Benchmarks, But Don't Buy Based On Them

Antutu keeps a running list of smartphone benchmarks, ranking them according to their performance. This is good information to have, but it should hardly be the only factor you consider. A phone that gets a good benchmark score might get poor battery life or have a crappier camera.

More importantly, the benchmark score only matters if you're planning to push your phone to its limits. If you use your phone for checking Gmail and reading the news, a crazy powerful processor won't do you much good. It might be better for playing mobile games, but that depends highly on how well game developers utilise that power. If you want to future-proof your phone, it might be a good idea to pick something in the top 20 or so, but don't sacrifice other features just to get a higher benchmark score.

It's also worth pointing out that the highest-ranking Android phone on Antutu's benchmark list is the OnePlus3T, which was accused of artificially inflating its benchmark score. While the mechanism they used to do this is complicated and it's not quite cheating, it also means that the score isn't necessarily reflective of everyday usage. This isn't the first time that a manufacturer has been accused of trying to tweak their system to get favourable benchmarks. This is yet another reason that you should take phone benchmarks with a grain of salt and not the foundation of your buying decision.

Skip the Hard Drive Benchmarks and Just Buy An SSD

Sites like Passmark and UserBenchmark offer comprehensive lists ranking both old school platter hard drives, as well as solid-state drives. Unless you're building a server for a business, however, you can ignore them. If you're using a platter drive, the best speed boost you can possibly get will be to get an SSD. If you already have an SSD, you're better off buying based on how much data they can hold or how reliable a particular model is.

Some old school platter-based drives are faster than others (for example, 10,000 rpm drives will be better than 7200 rpm ones), but SSDs don't use platters at all. This makes them much faster. Almost twice as fast as regular HDDs, in fact. If you have an older style hard drive, don't bother with benchmarks. Just get an SSD. If your computer is still slow, you'll probably see a bigger benefit by upgrading your other components.


Depending on your needs, you might care more about benchmarks than the average person and that's OK. If you're building a high-performance NAS or buying a phone to write apps on, you might want to pay more attention to benchmarks. However, in general, benchmarks are a useful tool that you shouldn't always treat as gospel.


Comments

    Benchmarks can be handy for a couple other reasons. Lots of benchmarks are also considered burn in tests. If you're building a new PC (and especially if you're overclocking) it's a good idea to stress your PC to make sure it's working properly. Benchmarks (and burn in tests) are good for this, if your PC crashes or fails a test or gets weird scores it's an indication that something is wrong. Maybe you need a bigger power supply or better cooling (or the heatsink on the CPU isn't quite mounted properly).

    It's better to do this test immediately after building (or overclocking) your PC so you can test and get a good idea if it's "right" straight away. Otherwise it could be weeks or months before you realise something is wrong - maybe you're getting random intermittent problems you can't pin down.

    The other reason benchmarking can be handy is testing changes. Yes benchmarking is a bit of a mysterious art, but it's often useful to get baselines on performance. Benchmark your new PC with a clean windows install. Then later on if you're thinking "gee my PC feels slow" you can run the benchmark again and compare. Maybe it is slow and the benchmark helps pin down what's wrong. Or maybe the benchmark is the same and it's just your imagination.

    And finally, remember not to try comparing your benchmarks with other peoples, or rather be smart if you do. Even identical hardware will have minor differences so just because your laptop is 1% slower than another identical one doesn't necessarily mean something is wrong with yours. If on the other hand it's 20% slower it's worth looking into.

Join the discussion!

Trending Stories Right Now