Firefox the Clear Winner for Automated Testing

As you probably know, Sauce Labs offers a tool for running automated tests an arbitrary OS/browser combinations. Inspired by Arstechnica’s breakdown of browser popularity among users, here’s our data on browser popularity among testers:

Tests by browser

Firefox has consistently been the most popular browser among testers. We used to think this was because it’s our default offering, but we now know that’s not true. Here’s why:

Firefox by version

As 3.6 aged and testers moved away from using it, Firefox’s overall slice of the pie remained the same size.

The popularity of each Firefox version is roughly related to how long it spent as the newest version and how recent it is. This is probably due to a tendency of some testers to write tests for whichever version is current and then not updating them.

IE by version

Surprisingly, testers seem not to care about IE6, the most difficult browser to support. Possibly this is because it’s also the most difficult browser to test; it’s notorious for crashing during tests, creating false positive test failures. Other than that, IE version usage looks the same for testers as it does for users; the old versions never die, in contrast to recent events in Safari testing.

Safari by version

Safari 5 took off in April 2012. This doesn’t reflect an overall increase in use; Safari popularity remained constant throughout this period. It simply cannibalized the use of Safari 4, in a more pronounced version of the Opera usage pattern.

Opera by version

Opera 12 is also available, but use didn’t start until after these numbers were captured. Testers appear to have stopped caring about supporting multiple versions of Opera, and are generally content to test with the latest version only.

Notes

• Google Chrome isn’t broken down by version, because we support only one version at a time
• Graphs based on 1,000,000 tests, randomly sampled out of 20,000,000 total
• Firefox 3.6 still accounts for a disproportionate amount of FF tests.  Our getting-started guides all use Firefox 3.6, and that’s the default browser-version combination. We thus can’t tell the difference between tests that wanted FF3.6 and tests that didn’t care what OS/browser they wanted, so you may still think it’s unfair to count a FF3.6 job as evidence of Firefox’s popularity. So here’s what the Firefox and overall graphs look like without FF3.6. Firefox is still the winner in the end, but only passes IE in late 2011 as testers move away from 3.6: