Browser Speed Tests: The Windows 7 Results

Firefox 3.6 Beta 1, like every other browser, makes a claim to being "faster". We took Firefox and all the other latest browsers, put them on Windows 7, and ran them through our human-measured speed tests to vet the bragging.

We've done a good number of these tests now, and the methodology remains much the same here — testing how long it takes for browsers to start up and load pages, and how much memory is eaten up from a user's perspective. We don't use a fancy multi-protocol benchmarking suite, mostly because each suite is subjective to developers' preferences and recording errors.

Browser start-ups are measured from double-clicking to the load of a locally saved Google home page, as "cold", or right after reboot and "warm", with the browser already having run once. Each browser is given a folder full of nine sites — up from eight in previous tests — and forced to load them all at once. Those timings are measured with Rob Keir's timer app, and done three times each and averaged, with way-out results discarded under the assumption of general computer wonkiness.

JavaScript testing comes from Mozilla's Dromaeo suite, which itself incorporates Google's V8 testing suite, Apple's SunSpider and a handful of independent JavaScript tests. We initially only ran the JavaScript tests in this first round, but Chrome's just-released Chrome beta makes a point of its prowess with DOM scripting, or the ability to use jQuery and other web scripting to alter page elements on the fly. We'll try to add in DOM test scores later on.

The memory results come from Windows' own Task Manager. In the case of browsers that use multiple processes, a screenshot is taken of the processes screen after the browser "settles", and a sum is added up there. In Chrome's case, we used the about:memory report, as Chrome/Chromium reports its memory a bit differently. The memory used by the about:memory tab itself is subtracted from the total reported memory.

Finally, these tests were done on a Lenovo ThinkPad T61p with 2GB of RAM and a 2.0 GHz Centrino Duo processor. For the first time since we started these tests, they were also performed on a fully updated, freshly installed final edition of Windows 7 Home Premium, rather than a fresh copy of Windows XP Professional. If you wanted to see how these browsers fared on Windows 7 versus XP, our last batch of speed tests compiled aggregate results for all browsers, with the same exact tests, minus the specific Chrome memory measurements.

Enough with the rules and regulations — here's how the browsers fared in our latest round of slightly obsessive-compulsive ratings. Note: Click any of the images below for a much bigger, clearer view.

Boot-up and page loading — Winner: Google Chrome (Development/Beta)!

In our last go-round, Opera's final 10th edition was just on the edge of beating Chrome's "stable" 2.0 release at cold and warm start-up times, while Firefox 3.5 tried its gosh-darn best. Either Chrome's development version made some huge changes between 4.0.203 and 4.0.223, or adapted better to Windows 7 or Chrome 2 bulked up a bit, because Chrome Dev consistently started up faster than its stable brethren. Meanwhile, Firefox 3.6 beta 1 bore out its claims to boosting start-up times, being nearly neck-and-neck with Chrome's stable edition, and Opera continues to be an impressively snappy browser.

When it came to loading nine tabs at once — all the browsers' home pages, plus Google.com, YouTube, Lifehacker and Gizmodo — Firefox 3.6 beta 1 continues to almost line up with Chrome's stable release, while Chrome development took the prize. As with our last tests, Internet Explorer 8 was surprisingly quicker at multi-page loading than Safari 4 or Opera, and truth be told, faster than Firefox 3.5 and nearly on-par with Firefox 3.6 or Chrome 2. It's installed and semi-running from start-up, so you might think it has an inherent advantage in the newest Windows release — but then again, look at how long it takes to start up.

JavaScript — Winner: Google Chrome! (For Now)

As mentioned above, we're planning to implement DOM/query-based testing into this test to give a fuller picture of how browsers manipulate code and move page elements around on the fly. On a straight-up test of JavaScript prowess on Mozilla's testing suite though, both of Chrome's versions seem far ahead, Safari earns a sure second place, and the rest are fairly matched (Internet Explorer 8 refuses to finish on Dromaeo, so we left it out of this chart).

We'll note here, as many browser developers have, that pure JavaScript speed tests can be considered akin to horsepower engine tests — they don't tell you everything about performance and might be adjusted for by browsers' JavaScript engines.

Memory use — Winner: Firefox 3.6 beta 1

Firefox 3.6 beta 1 continues Mozilla's mission of reigning in memory use, both at start-up and with content loaded, with even slimmer returns than 3.5. Opera 10.01 was a little slimmer than Safari this time around on Windows 7, and Chrome, measured with its own memory counter, still takes a fairly hefty commitment to pull off that speed. Then again, if you're rocking a laptop with 4GB memory, that might not matter so much.

Grading the results

We're going to try something a little new here, assigning a number grade to each browser's performance in the various categories. They'll be based on the raw numbers, but the ranking assigned to them is our own doing.

Given seven browsers competing in each test, we'll rank each browser's performance on a scale of 1 to 7 (7 being the best performance, 1 being the worst), and awarding a tie when the results are close enough to be affected by one editor's timing reflexes. After totalling up the point scores and dividing by the total number of points available, here's how they fared in each test, to our eyes:

Cold and warm start-up times

  • Chrome 4.0.223.11: 7
  • Opera 10.01: 6
  • Chrome 2 (stable): 5
  • Firefox 3.6 beta 1: 5
  • Safari 4.03: 4
  • Firefox 3.5.4: 3
  • Internet Explorer 8.0.7600: 2

Nine tab load

  • Chrome 4.0.223.11: 7
  • Internet Explorer 8.0.7600 6
  • Chrome 2 (stable): 6
  • Firefox 3.6 beta 1: 6
  • Firefox 3.5.4: 5
  • Safari 4.03: 4
  • Opera 10.01: 3

JavaScript

  • Chrome 2 (stable): 7
  • Chrome 4.0.223.11: 7
  • Safari 4.03: 5
  • Firefox 3.6 beta 1: 3
  • Firefox 3.5.4: 3
  • Opera 10.01: 2
  • Internet Explorer 8.0.7600: N/A—counting as 1

Memory use

  • Firefox 3.6 beta 1: 7
  • Firefox 3.5.4: 6
  • Opera 10.01: 5
  • Safari 4.03: 4
  • Internet Explorer 8.0.7600: 4
  • Google Chrome 4.0.223.11: 2
  • Google Chrome 2 (stable): 1

Averages

  • Google Chrome 4.0.223.11: 82.1 %
  • Firefox 3.6 beta 1: 75%
  • Google Chrome 2 (stable): 67.9%
  • Firefox 3.5.4: 60.7%
  • Safari 4.03: 60.7%
  • Opera 10.01: 57.1%
  • Internet Explorer 8.0.7600: 46.4%

Conclusive scientific results? Not so much. Entirely objective? Not when it comes to our grading scheme, or the speed of one editor's eyes and hands. That said, we're glad to see measurable improvement in Firefox 3.6 beta 1, and we're always eager to see what Opera, Chrome and Safari have been building into their latest releases. Internet Explorer, hey, you just keep keepin' on, yeah?


Comments

Be the first to comment on this story!