The Surprising Worst Browser

August 17th, 2011 by The Sauce Labs Team

We have lots of data

Just like any good software company, we track all kinds of data. Last time we shared some of it with you, you loved it, and that data wasn’t even the good stuff. We thought it was, but it dawned on us recently that we’ve been indirectly been tracking something better. And when we realized where to look, we found something unexpected.

In case you haven’t heard, Sauce OnDemand is a tool for automating real browsers (try it out! it’s awesome). We have metadata about millions of browser sessions our customers have used to test their actual websites. As everyone knows, sometimes your software doesn’t work. Maybe it crashes or maybe you had a bug. Selenium testing on Sauce is no different – almost 100% of the time, nothing goes wrong; our reliability in the last few months is at least 99.94%. In fact, as you’ll see later, we’re now more reliable than modern browsers. But sometimes there’s an error that we think may have been our fault. When there is, we refund the customer and work to fix it. We also record that there was an error. Does that seem significant enough to be italicized? If it does, you’re smarter than we were.


We know which browsers cause the most errors

See, sometimes job errors were caused by connectivity, or bugs in our code, or maybe neutrinos from outer space. But some of the time, they were the browser itself crashing. For each error, it would take real investigation to figure out what caused it, and we have thousands of them. But our code and our customers’ code is independent of the browser being tested (which is the whole point of both Sauce OnDemand and Selenium), so if we only look at relative error rates broken down by browser, we can see which browsers are least reliable. Nobody else has this data. This is the *only* statistically significant study of browser reliability on real webpages. Check for yourself – it’s not out there.

Browser fight!

Error rates (percent) by browser and version*

The numbers in the graph above are misleading for a some of the browsers. Here’s why:

  • We stopped supporting Firefox 3.5 a while ago. That means all FF3.5 jobs are from a long time ago, before we’d had much time to streamline and errorproof our system. All browsers would have a higher error rate if we only look at jobs from a long time ago.
  • IE9, FF4, FF5, and Opera 11 are new. This is the opposite of the FF3.5 issue. All jobs run recently have lower error rates because our service continues to become more reliable as we fix bugs our users discover. These browsers’ jobs were all run recently.

Fair browser fight!

This is the same graph as above, with the unfairly advantaged or disadvantaged browsers removed


Error rates (percent) by browser and version*

Yeah, IE6 sucks

If you’re tech savvy, most of these results aren’t very surprising.

  • IE6 is one of the worst browsers. Each newer IE is slightly more stable but still not good.
  • Firefox is solid.
  • Google Chrome is the big winner overall. They force you to update to their newest version every session, so their error rate is an average across all their versions, but it’s still significantly better than even the newest versions of the other browsers.
  • Opera is fine.

The shocker is Safari story. Safari 3 – the oldest version of safari – is extremely reliable. Safari 4 is a good deal above average. And then there’s Safari 5.

The Safari 5 surprise

Safari 5, the latest in browser technology from the most valuable company in the world, is by far the worst on the market. Go have another look at that number – it’s almost twice as bad as second-worst, the oft-maligned IE6. And that comparison is unfair to IE6. See, Safari 5 was released recently, like Opera 11, IE9, and FF5. Those are the browsers whose error rates were unfairly good. Like them, all Safari 5 jobs were run on our newer, ultra-stable OnDemand infrastructure. We should expect it to have an extremely *low* error rate, like they do, but instead its error rate is ten times worse.

At first we thought the high error rate could be the result of the fact that we always run Safari on Windows, while it’s made by Apple.  That’s easy to dismiss, because earlier versions of Safari were fine. Then we explored the possibility that the errors were caused by always running Safari 5 in proxy mode (for arcane Selenium reasons). So we looked at the average error rate for non-Safari jobs run in proxy mode, and it put things back into perspective.

Proxy mode browser fight

Error rates (percent) by browser and version*

These are the error rates for browsers running in proxy mode (we don’t have enough data for Opera). Notice the new scale on the Y axis.

Surprise! The worst browser wasn’t a surprise after all

As you can see, Firefox actually seems to perform better in proxy mode, so we can’t say that proxy mode is always worse.  Chrome is unaffected. Safari 5 is still the worst Safari, but it’s no longer a huge outlier overall. IE7, the best IE in proxy mode, is on par with the worst Firefox, 3.6. IE8 is surprisingly much worse than IE7. And the king of being a bad browser, once again, is IE6. Hail to the king, baby.

*Percents are 1 – lower bound 95% confidence Wilson score for success rate

Comments (You may use the <code> or <pre> tags in your comment)

  1. Rich Morin says:

    Are you sure there isn’t something about the way you’re running Safari that is causing it to fail? I’m not sure what this would be, but I could imagine some sort of API or timing issue. If so, your results would not necessarily translate to real-world behavior.

  2. Can Duruk says:

    Interesting data indeed. As a Mac user — who doesn’t have Flash installed –, I’ve always found Safari very reliable. What is the host system used for these tests?

    Also, in my limited experience with Selenium, it had its own way of communicating with the browser. Doesn’t that make your tests a not so representative of real-world use?

  3. Anonymous says:

    Firefox 3.6 is outdated now, by the way.

  4. Brad says:

    @Anonymous Yeah, and IE6 is from 2001, but they’re still testing it. Because a tiny fraction of the world still uses it. More people use FF3.6 than IE6.

  5. nrcha says:

    @Brad As sad as it may be, the reality is very gloomy, if you are browser-based app developer. Most big companies are very reluctant to upgrade their systems. If-fact, they will do everything to avoid upgrading, because quite often they have some app, that is already “optimized” for ie6. Often there are another reasons why upgrading is not an option, or, is even a “hazard”.

    And yes, IE6 sucks :)

  6. John Peyton says:

    Please change your CSS so that there is a margin of at least a few pixels on the left side when the browser is sized to a reasonable width.

  7. CTW says:

    Safari is the most overrated browser on the market and I cringe when it comes time to support it. The mighty Apple still hasn’t learned the trick of creating compelling user experiences in and around the browser.

  8. bella says:

    i am a user of XP.I want to use ie9 but you know it doesn’t support XP.So i turn to avant browser.better than ie and easier to use .i have a question,who can give me a beautiful skin for avant browser ?his skin….

  9. [...] anche le ultime versioni di Safari non è che se la passino poi così bene ( vedi quel che ne dice sauce labs [...]

  10. Brad says:

    If the U.S. Federal government is your client, then you will likely be coding for IE6.

  11. Brad says:

    If the U.S. Federal government is your target client, you will likely be coding for IE6.

  12. wr says:

    I realize that it is a universally accepted and unquestioned truth that all Microsoft software is terrible, and Internet Explorer the most terrible of all. But given that the score for Firefox 3.6 is 0.2 and the score for IE8 is about 0.21, how do you conclude that “Firefox is solid” but “Each newer IE is slightly more stable but still not good”? That seems like a conclusion that everybody *knows* to be true, whether or not it is backed up by the data

  13. Russell says:

    Where’s IE9 in those graphs at the end?

  14. Whatever says:

    So you stopped supporting Firefox 3.5 “quite awhile ago” when it was only released in the Summer of 2009 (OMG! It’s TWO YEARS OLD) but still track numbers for a 10 year old MS browser?

    Firefox 3.5 has about 4X the usage over IE6.
    Maybe you’re a bit obsessed?

  15. Jonathan says:

    I think what needs to be clarried here is ‘What is an error’. Would it constitute a Javascript function not executing correct? Would it be the browser crashing? Or is it how the page renders?

    What exactly are the guidelines used defining the error standard

  16. Denzien says:

    @wr Clearly, FF3.6 is shown to have a 0.195% error rate and IE8 has a 0.22%. This is WORLDS apart, thus invalidating your insinuation

    [/sarcasm]

  17. Izzy says:

    “Each newer IE is slightly more stable but still not good. Firefox is solid…. Opera is fine.”

    WTF!? Let me get this straight… Firefox is “solid” with 3.6 at ~0.19 while IE is “still not good” with IE8 at ~0.23? Opera 9 and 10 are somehow “fine” when they appear to have a nearly identical profile to IE 7 and 8?

    It’s a good thing you included the graph, because your personal prejudice is screaming through.

  18. zenstain says:

    @wr .NET does not suck at all, let us target our MS criticisms to the most deserving products please. That said, oh the IE CSS fun I used to have… :(

  19. Have you considered the platform you used (Windows, Mac, Linux, etc.) with the tests? I agree with you sometimes Safari 5 fails, though not always, when I do hardcore processing in Javascript. I tried once developing on cross-browser and cross-platform to test efficiency on sites I developed. Safari tends to fail in Windows while it doesn’t on Mac (we used a virtual machine running mac with the same computer used on testing Safari on Windows).

  20. RiskFactor says:

    OK, so the measure of a stable browser is all about proxy mode! I never knew!

  21. ScottK says:

    Proof that if you massage data enough, you can get any result you want. There are liars, damn liars, and statisticians. (Not that I’m calling you liars, per se, just sayin’…) :)

    And I noticed a fairly significant bias shows up pretty much right from the start: IE 7 and 8 have a rate almost on par with Opera – by eyeballing, a difference of maybe .005 to .01; Firefox 2 and 3.6 have rates just slightly better than Opera, maybe .01 or so. Yet IE 7/8 is “not good”, Opera is “fine” and Firefox is “solid”. I’m not defending IE, just looking to your objectivity.

  22. karl says:

    You seem to say that Opera 11 is new. In fact Opera 11.50 is the current released stable version. So not really new. Opera Next (or Opera 12 Alpha is the experimental version). So if you really want to be on par with other browsers you would have to include Opera 11.50

  23. john says:

    I’m with ScottK, IE8 is either even with or slightly worse than opera 10 and defintatly between opera 9 and 10, why is Opera “fine” and IE “not good” its also only slightly worse than FF which is “solid”. Also, before someone thinks i’m a MS fanboy, I hate IE 6, 7, and 9(quite a few pages cause the tab to crash). IE 8 is useable, but i’m switching to chrome.

  24. Dan Sutton says:

    I agree that Safari is a disgusting heap of bugs… I actually like programming for IE: in Javascript, it seems to know what you mean a lot better than most other browsers. My least favourite is Firefox: it seems to get things wrong deliberately, and I always have the feeling when using it that I’m somehow under water: everything seems to be very second-hand. Opera is an attempt at a browser, but it’s really annoying that it doesn’t support oncontextclick (a massive omission) and it renders things badly. Chrome is excellent.

  25. mike says:

    You people are trying to sell a service? As well as your biased evaluations(IE8 v FF3.6 and Opera10), your data is virtually meaningless. As “more reliable as we fix bugs our users discover” shows, this speaks almost zero meaning. What, you went from 90 errors to 85 to 83? Your errors are part of it!?
    Bah

  26. anon says:

    I think your “data” speaks more for your crap software that you write than it does for the browsers. Who told you that you were a “good software” company? Your parents?

  27. scorcher says:

    I completely agree with the last couple of comments…

  28. sosho says:

    this the stupidest article ever…

    so FF4, FF5, IE9, O11 are all new browsers and it’s unfair to include them in you $hity comparison where as the updated Chrome is an old browser , get real …

    FF3.6 updated to FF4 then FF5 and now FF6 (still there the upcoming FF7 & 8)
    O10 updated to O11
    IE8 updated to IE9

    [Quote]
    Google Chrome is the big winner overall. They force you to update to their newest version every session, so their error rate is an average across all their versions, but it’s still significantly better than even the newest versions of the other browsers.
    [/Quote]

    Google Chrome is the big winner, others are losers ::this so stupid…

  29. morego says:

    The tests would be a lot more useful if they applied to something say within the last year or so – like recent versions of these browsers.

  30. FX says:

    Doesn’t even test Firefox 6.0 Final, Firefox 7.0 Beta or Firefox 8.0 Aurora. All outdated Firefox browsers, really?

  31. [...] The Surprising Worst Browser (saucelabs.com) Posted in CSS, HTML, Web Design [...]

  32. Daniel says:

    If you are a web developer, you will know all of the errors related in making a page work in the browser. Every other browser is fine, but IE completely crews up the format and everything. It is by far the worst browser for developers and supports so little of the code that Firefox and Chrome do.

  33. nVrness says:

    This might be the most useless and well just plain wrong ‘test’ or whatever the hell it’s supposed to be.

    I’m not sure what you were doing or how you conduct your ‘tests’ but clearly it’s a fail of epic scale.

    You should really go find other employment.

  34. [...] us quite a lot of data about browser reliability when it interacts with Selenium. Back in 2011, we took a look at that data to find out which browsers had the highest error [...]

  35. [...] error rate for non-Safari jobs run in proxy mode, and it put things back into perspective,” Sauce stated in a blog [...]

  36. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  37. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  38. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  39. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  40. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  41. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  42. [...] The good news, at least for Mozilla fans, is that Firefox’s error level was so low that it achieved a remarkable zero. In other words, so close to zero that it can be said that it had no errors at all during the tests. [...]

  43. rain says:

    Salsa Labs claims that it supports “over 2,000 user groups’ relationships with over 40 million supporters, members, donors, activists, and fans all around the world.”

    -wikipedia

    I cant find anything else out about this Company, nor its “data”, theres no link to the data or what was tested.

    I find a bunch of other cross-browser test programs :
    Selenium, utest and testingbot and so on.

    last time i saw a browser test in a Magazine, they used Acid, and that one was pretty well documented.. added bonus, it was free. and one could check the testing results live.

  44. [...] looks like another stick to beat Microsoft’s browser with, it is worth pointing out that the firm’s study of around 55 million tests run using the Selenium platform found that all browsers showed extremely low levels of errors, in [...]

  45. Dylan says:

    Salsa Labs sound like great people, but they’re not actually related to us; We’re Sauce Labs, we provide over 150 platforms to test web and mobile applications on. We actually use Selenium for our product, you can find details at Our main site. We got our data from logs of tests our clients ran, where the browser crashed, the test aborted prematurely or something else went “wrong”.

    We’re not testing individual browsers against features or rendering or compliance with standards, just aggregating statistics about which browsers didn’t complete their tasks for some reason, across our millions of tests. ACID and its kin are more about feature tests for individual versions of individual browsers, whereas this data is about stability of individual browsers, given they’re being automated with Selenium.

Leave a Comment