The Best (and Worst) Browsers to Test With

August 20th, 2013 by lauren nguyen

We run a lot of Selenium testing on Sauce. And with those tests, we see a lot of failures. Sometimes failures aren’t due to bad code, but to the actual browser crashing. Thankfully, when this happens while running a test, we have measures built into our system that reboot the browser and rerun your test for you. This happens without you even knowing, and helps to make your testing life easier.

But we still keep records of every time a browser crashes on our service. We’ve done this for all the tests ever run on Sauce, which gives us quite a lot of data about browser reliability when it interacts with Selenium. Back in 2011, we took a look at that data to find out which browsers had the highest error rates.

Two years later, with even more data (and more browser versions), we decided to take a look at the information again to find out: Which browser has the highest error rate?

Error rate by browser

We queried for all tests ever run on Sauce – 55 million and counting – and looked at the error rate by browser. The results were, well, not surprising.

Yes, it turns out Internet Explorer performs the worst among our users’ tests, with a .25% error rate. But there’s more to the whole story here. This graph shows the error rate of all browser versions grouped together, but each browser has many different versions with varying error rates. So we broke down the relative error rates by browser version.

Error rate by browser version

Below, you can see the browser versions with the highest error rates that have at least 1,000 tests run on them in our service.

 

When we look at error rates by browser version, IE 6 & IE 7 are the clear winners (well – losers), with error rates of .31% and .29% respectively. Coming in behind IE 6 & 7 is Chrome 5 followed by Firefox 3.5. Only one later version of Chrome version appears in the graph, Chrome 19. Most versions of Chrome have error rates low enough that they didn’t show up on the graph. Neither do later versions of Firefox.

But all of the browser are improving with time, so we then decided to take a look at the error rates for the latest versions of all the browsers that we had data from at least 1,000 jobs on.

As you can see, for recent browser versions, Safari 6 has the highest error rate in our service at .12%, which is less than half of IE 6’s .31% error rate. Chrome 27 and Firefox 22 have virtually nonexistent error rates, with IE and Opera falling in the middle.

Opera has a surprisingly high error rate compared with other recent browser versions. And while Safari 5 has a .16% error rate, Safari 6 has fallen to a .12% error rate, which overall isn’t horrible.

Half of the browser versions we analyzed had error rates lower than .07%. That’s pretty low, and suggests that browsers are getting more reliable as more versions come out.

IE – Getting Better All the Time

Finally, we looked at a graph of error rates of different versions of Internet Explorer, and found an interesting trend.

Error rates for IE have been getting significantly lower with each new version. Microsoft has been so good at squashing their historically high error rates that IE 10 has an impressive 0.05% error rate.

Microsoft has really been pushing their new version of Internet Explorer as the most modern and high-performing version of IE. You’ve may have seen their nostalgia-tinged IE 10 commercial. The data we have on browser error rates suggests that their claims may have quite a bit of merit.

Though browser usage share is an imperfect science, indicators would suggest that Chrome is giving IE a run for its money. But will Internet Explorer’s changes and improved performance give them what it takes to climb back to the top? Guess we’ll have to wait and find out.”

Comments (You may use the <code> or <pre> tags in your comment)

  1. [...] a research conducted by Sauce Labs reveals that Internet Explorer is still getting more crashes than its main rivals, including here [...]

  2. [...] there is still a lot of room for improvement, as a research conducted by Sauce Labs reveals that overall, Internet Explorer is still more prone to crashes than [...]

  3. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  4. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  5. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  6. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  7. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  8. edward.h says:

    hi,
    awesome to see the interesting crash report between browsers.
    but which let me felt curiousness is
    “how to do the browser steady test”
    it looks like
    “let the browser visit many websites,and record browsers’ crash case”?

    can i know your method in browser’s steady test?

  9. Anonymous says:

    Not that I contest these numbers but an interesting fact would be the primary browser used during development.
    If you develop everything on Firefox 22 you are obviously going to have a very low error rate.
    Even more interesting would be the same analysis after forcing developers to use IE as primary browser.

  10. opera eng says:

    The latest Opera version is 15 not 12

  11. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  12. Simon says:

    I’m Glad to see Microsoft actually improving Internet Explorer over time.

  13. Damian Kumor says:

    Opera went from 12 to 15 a month ago. The next one seems very stable to me since it’s using Chrome as it’s core.

  14. Nick says:

    What are these? Chart labels for ants!?

  15. Sam Tuke says:

    Are these error rates not due to the quality / support of the respective selenium drivers / integration in the browsers?

    Maybe Firefox has best Selenium support and this has the lowest error rate, rather than it simply being the most stable browser.

    I’d be very surprised if Opera was really as unstable as these graphs indicate.

  16. [...] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post [...]

  17. [...] were no major releases during the month of August. Sauce Labs did release a report on August 20th saying that IE had the highest error rate (crashes) among browsers, while Firefox [...]

  18. Howdy just wanted to give you a quick heads up.
    The words in your article seem to be running
    off the screen in Chrome. I’m not sure if this is a format
    issue or something to do with browser compatibility but I thought I’d post
    to let you know.The design and sthle look great though!
    Hope you get the issue fixed soon. Thanks

  19. Dylan says:

    It can be hard to disentangle those statistics; I’m personally an Opera user and find it fairly stable, having said that I barely ever see a browser crash from Chrome which I also use daily.

    Ultimately knowing what kind of failure rates you get for automation is the most useful when considered in light of automation; But it’s fun to look at them, and to maybe play with some expectations a bit.

  20. […] newer versions show much lower rates of troubles bad enough to crash the browser, according to a Sauce Labs blog post […]

Leave a Comment