How We Test Web Browsers

Lifehacker regularly runs browser speed tests to see which browser runs the fastest in real-world situations. This is how we currently test browser performance.

We run our tests on a desktop PC with an i7 processor overclocked to 3.8 GHz and with 12GB of RAM. The machine has a wired internet connection, which eliminates the unreliability of Wi-Fi. As such, our numbers very much represent a best-case scenario. The numbers on your machine will differ, but the relative performance differences are likely to remain.

Start-Up & Page Load Times

For the time-based tests, we measure “cold” start-up (just after reboot, with the browser not having run yet). We also test a browser’s ability to load multiple tabs at once (our test group of sites includes Lifehacker, Bing, Hulu, Amazon, Wikipedia, Facebook, MSN, YouTube and eBay). Each of these tests is measured by hand, starting a timer at the same time as the mouse is clicked for that specific task (such as launching the browser from the taskbar or opening all nine tabs at once from the bookmark bar).

At least three tests are done for each measurement on each browser, and an average taken from those three. Any obvious outliers (as in, 2.8 seconds, 3.2 seconds, 7.9 seconds) are removed and replaced. We test from the moment we launch the browser to the point where the browser window shows up for cold starts, not when the page shows up, since many people will start using the browser (clicking bookmarks, typing in an address) before their home page even appears. For the tab loading test, we wait until every page has loaded and the wheel on each tab has stopped spinning.

JavaScript & CSS

JavaScript and CSS performance are an increasingly important metric for browsers, as webapps become more powerful and more of our work moves into the cloud. Currently, we use Mozilla’s Dromaeo testing suite. No testing suite is perfect — browser makers can “write to the test” and today’s measurement metric might not provide an accurate picture of what makes tomorrow’s browser feel so snappy. Dromaeo combines tests from the two leading JavaScript proponents (Google and Apple), so it’s a reasonable if not perfect alternative. (There’s no evidence to date that Mozilla’s authorship of this testing suite has given Firefox any obvious advantage in the results.)

Memory Usage

To measure memory, we use Google Chrome’s about:memory function. As more browsers adopt multi-process architecture for stability and security, Windows’ measurement of memory used by browsers has become inaccurate. Chrome’s tool reports on itself, but also every other browser we’ve tested. (When measuring Chrome’s own memory usage, we subtract the memory used by the about:memory tab itself, to ensure it doesn’t artificially increase Chrome’s score.) We also let each browser sit for a while and wait for its memory usage to level out before measuring, since most browsers will initially keep accruing memory usage after loading.

Extensions

Most Lifehacker users will rely on extensions, so testing with them present is important. To measure the impact of extensions on memory use, we run our memory tests with five representative extensions for Chrome, Firefox and Opera:

• Xmarks bookmark sync
• LastPass password manager
• AdBlock Plus
• Gmail-based mail checkers: Gmail checker for Firefox, Google Mail Checker for Chrome, and Gmail Checker on Opera
• InvisibleHand bargain hunter