Over the last year, the Sauce customer base has been growing tremendously. We have heard your concern about our recent service interruptions. We have always had our sights set on 99.99% uptime, and are continually doing everything we can to achieve that. Examples include our recent datacenter move, constant capacity upgrades, and complete re-architecture of our Mac cloud.
Unfortunately, our growth has caused some hiccups that have negatively impacted your experience using Sauce. Knowing ourselves how important a smooth development process is to productivity, our recent quality of service is simply unacceptable and we vow to do better. We want it to be 100% clear that reliability is our top priority, and every time we can’t run jobs within single digit seconds, it becomes a red alert of the highest priority for our team.
To give you insight into how we are assessing situations such as the one yesterday: when our average launch time for a job exceeds 30 seconds, we consider that event an outage and launch an investigation to fully understand what happened, and why, with the onus on not only fixing the problem, but improving our process going forward. These include such things as eliminating single points of failure, scaling our cloud capacity far beyond the foreseen needs of our users, and introducing more stringent testing and analysis into the development and pre-deploy processes.
In terms of communication, when an outage happens we update @sauceops immediately and also status.saucelabs.com, which shows the average wait time for jobs. Our status page is continuously monitoring the situation and updating every few seconds. We also take care to update @sauceops when we notice that job wait times are exceeding an average of 10 seconds, though we don’t count that as an outage.
Additionally, we’d like to make a suggestion to help isolate your build from the effects of unexpected elevated wait times and internet latency should they happen in the future. If you are using a CI system or test runner that times out and fails tests, please consider increasing your time-out threshold to five minutes to isolate your suite for when the Sauce Labs cloud experiences longer wait times. This will allow your jobs to remain queued rather than fail on Sauce. Again, you can go to http://status.saucelabs.com at any time to see average wait times and tweets from @sauceops.
Rest assured that we take events like yesterday extremely seriously, and are working to prevent this from happening in the future. We are fortunate to have very loyal customers and don’t want to let you down.
Today we’re excited to let you know that you can now test your hybrid and native apps with iOS 7 on Sauce. With more than 200M downloads since its release, we know that this is an important platform for our users to test on.
To get started, visit our browsers and platforms page for Appium and copy and paste the DesiredCapabilities provided for either version of iOS (or Android) you want to test on. And if you’re new to mobile test automation in general, check out our Getting Started with Appium guide.
Ever run into an error message on Sauce you don’t understand? Alex put this handy list of error messages on Sauce together. You can find the original doc here. This will soon be available for reference on Sauce’s website, but in the meantime, brush up on the error messages you might see on Sauce and what they mean.
==== AUTOMATED JOB ERRORS: ====
– Invalid credentials –
Some combination of the following error messages will be thrown:
OpenQA.Selenium.WebDriverException : Unexpected error. Unknown username.
You sent username ‘None’ in your browser string, which is not a valid Sauce Labs account.
OpenQA.Selenium.WebDriverException : Unexpected error. Invalid Credentials.
org.openqa.selenium.UnsupportedCommandException: Invalid Credentials.
You sent the access key ‘None’ but it does not match the access key associated with your account. Please login to saucelabs.com to retrieve a valid access key.
This indicates that Selenium could not parse the credentials you sent. If you’re using “access-key” in your test setup, try replacing it with “accessKey”, or vice versa, as it’s language-dependent.
– User Terminated –
This means that you ended the job, either through the “Cancel Job” button, by jumping into an automated session from the Sauce website, or via a breakpoint. Since this terminates the job immediately and hands over control to you, the test will not upload any screenshots, video, or logs that were collected previously.
– Timeout errors –
1. “Test did not see a new command for 90 seconds. Timing out.”
2. “Selenium took too long to run your command”
3. “Test exceeded maximum duration after 1800 seconds”
Sauce builds in a few default timeouts to keep uncontrolled tests from eating up your minutes. Here’s a brief rundown on what each of the available timeout settings does:
1. The “idle-timeout” (default 90 seconds) ends the job if no commands are received from your Selenium script.
2. The “command-timeout” (default 5 minutes) ends the job if a single command takes too long to run.
3. And the “max-duration” timeout (default 30 minutes) ends the job if your entire test takes longer than you want.
You can set these to different values using “desired capabilities” settings in the setup of your test, as described at the URL above.
– Unsupported OS/browser/version combo –
Check to make sure that the browser, browser-version, and platform settings you’re using are in our supported list.
And note that there’s a separate list for Appium tests (as they use a different mechanic to drive the tests).
– Browser failed to start –
The twin sibling of “Unsupported version”, this message means that something a little more unusual is off in your test setup. Usually, this means that you’re specifying a Selenium version that isn’t compatible with the browser/version/OS you’ve selected. (For example, you should not be setting this for any mobile tests.)
Try simply omitting that setting, and if you still see the issue, feel free to write to email@example.com with a description of the issue and a copy of your setup code.
– Client disconnected during getNewBrowserSession request –
This means that your test runner decided to end the job before it had fully initialized on Sauce’s end. There are a few potential causes:
1. You’re running too many tests at a time: Check the left sidebar on your Account page. It shows a “Parallel tests” number, which is the maximum number of tests you can run at a time, based on your subscription level. If your account can run 2 parallel tests, and you’re launching 10, 8 will be “queued” until one of your tests finishes and a slot frees up. However, if this takes a long time, your test runner may choose to end the queued jobs after a few minutes instead of waiting. Just make sure you’re launching a reasonable number of simultaneous tests for your account.
2. High job wait times: Check our status page and/or follow @sauceops on Twitter for up-to-the-minute news about any issues within the service. If something causes demand for certain VMs to stack up, your jobs may be queued and (as above) terminated by your test runner.
Tests that end this way are never taken out of your minutes.
– Runaway execution. Please contact firstname.lastname@example.org for assistance. –
This message means that an error has been detected in the VM or in Selenium, which caused the test to behave abnormally, and Sauce detected this and shut down the job. These are very rare and usually do not recur. If you do see more than one on the same test, let us know.
==== MANUAL JOB ERRORS: ====
– Job does not load –
There are two common scenarios here:
• Error message: “Uh oh! Some error occurred while connecting to the browser”
• The job seems to start, but you see only a white text box in the middle of a black screen.
Both of these indicate that your browser is having trouble displaying the VNC stream from the remote machine. Take the following steps to troubleshoot:
• Check the video on Sauce: If the recorded video after the job shows a steady video stream, this indicates that the issue is in your computer or connection to Sauce. However, if the Sauce video shows the same issue, that indicates an issue in our service. In that case, send us the URL for the job page and a screenshot of the issue.
• Check that your browser is up to date: If you’re on an older version, this may cause incompatibilities. Update your browser and try again.
• Check your firewall: make sure that your machine allows full access for the interactive stream over the required ports.
• Check that the Internet connection is stable: We recommend running Sauce tests from a machine with a wired Ethernet connection, to ensure a steady connection. If the connection flickers, this error could be thrown.
==== FRAMEWORK-SPECIFIC ERRORS: ====
– Appium: Failed to download mobile app –
First, if you’re having trouble, try using Sauce’s temporary storage. Uploading your app to our servers makes it download faster.
If you’re using this, make sure that you’re specifying the “APP_PATH” as “sauce-storage:your_file_name” — NOT the saucelabs.com/rest/v1/storage… URL.
And when you upload your file to our server, check the JSON response. It should be formatted like this:
For all of you who run manual test sessions with Sauce Labs, we’ve got some great news. We recently released our updated Sauce Launcher plugin for Google Chrome, with lots of improvements to help you run manual jobs more easily.
Sauce Launcher makes launching a manual test as easy as clicking a button, and we think the update makes it even better. We updated the plugin’s look, feel, and layout to better match saucelabs.com, in order to make it easier to use. Sauce Launcher now also offers all the browsers and platforms we support, and it now automatically updates when we add newly supported platforms and features so it will never go out of date again! Download it in the Chrome webstore, and let us know what you think. Happy testing!
Today we are extremely excited to announce the initial release of an important browser platform on Sauce: the classic — but innovative — text-based browser Lynx. While Lynx is not as popular as more modern, graphical browsers like Internet Explorer 6, it has a devoted following and a compelling feature set, and we feel it’s important that Sauce customers be able to test their web apps in a text-based browser.
Ever since Twitter demonstrated the power of simple, textual interfaces, we have been looking at Lynx as a bold way forward for the web. We think its less-is-more approach represents the future of sustainable web technologies. Because it spends less energy on graphics, Lynx probably has the smallest carbon footprint of any web browser out there, and lightning-fast performance.
And because Lynx is so different from other browsers, we believe it’s extra important that developers be able to test their web apps against it. That’s why we’re excited to extend Sauce Labs’ cross-browser functional testing cloud to include interactive and Selenium testing for Lynx.
We recently added a new feature to our manual browser testing web client that we’re excited about. The “Live Share” feature in Manual Sauce, our interactive testing tool, allows you to share your live browser sessions with coworkers simply by sending a link. You can collaborate using your favorite IM client to demonstrate the exact steps to reproduce a bug you’ve run into and interact with the browser running on Sauce at the same time!
To try this out, first start a manual testing session. Once your browser has loaded, look to the header bar for the live share button next to the URL bar. Hover over the button to pop open a dialogue box that lets you specify the email address of the person you want to receive a private, unique link for sharing the manual test session. Then click the share button.
If you leave the email boxes empty and click the share button, no email will be sent but your test’s visibility will change to ‘share’. This means anyone with the link can access the test session and view it live. If it’s easier to share the session directly via Gmail or whatever email client you prefer, just copy the link from the live sharing dialog and send it to your coworker(s).
To indicate that the tests can by accessed by anyone with proper link, the sharing button will be turned on next to the address bar. You can click this button to turn off live sharing mode. Each time you click that button, it makes a call to our REST API and the test visibility mode is updated (http://saucelabs.com/docs/rest).
Please note that by default, the job visibility mode is set to ‘private’. This way, only the owner of the test can access it. When you click the button, the mode is set to ‘share’, allowing everyone with valid secret link to access the test page.
We hope you find this new feature useful. And as usual, let us know if you have any questions. Happy testing!
Today I’m going to show you how you can use Selenium Builder with GitHub, Travis, and Sauce Labs to make a continuous integration solution for your website that’s all on the Internet, nothing stored locally. You can either follow along with the demo video below or read on for the written tutorial.
The great thing about this setup is that if you put your site code and your tests into the same GitHub repository, then whenever you update the site code, and whenever you update the tests, Travis will rerun your Selenium tests on Sauce OnDemand – and then send you an email about whether or not they still work!
So you can get a full Selenium CI solution by putting together these components. And because they’re all components, if you want to use something else in part – for example Jenkins instead of Travis – you can construct your own solution.
For now I’m going to show you this set of four things that work really well together. The great thing about this is that you store everything on GitHub, so all your tests are stored in the cloud, nothing is stored on some local machine somewhere. You can record and edit tests from any computer that runs Firefox. You can access your tests, edit them, & play them back locally. And all of the actual execution of the testing – the CI – is handled by Travis and Sauce. So you actually don’t need to build any of your own testing infrastructure!
OK! So how do you actually do this?
The first thing you need to do is install the plugins. You go into Selenium Builder, you go into Plugins, and you just click install on GitHub integration and Sauce. Restart Builder.
Next up, you want to record some tests. Let’s just say for now we’re testing this website.
Let’s record a verification.
Now we’re going to save this test. In the File menu, save the test to GitHub.
Because this is the first time we’re using it, I’m going to have to enter my details.
Now, as you can see, we can access our repositories. And I’ve set up this little repo here as an example one.
So let’s save this test in here, as JSON, which is the standard format Builder works in. This means we can later open it again and edit it.
Let’s do another test.
Now we save that one as well. Just record as many tests as you want and put them all into this place.
Now we’ve done that, the next thing we need to do is configure the Selenium interpreter, which is actually a little node.js program that takes these JSON scripts and uses Webdriver to play them back.
Let’s clone the repo to disk so we can add the config files we need.
Create a new text file. I’ve just pasted in a bunch of things, which are actually reasonably straightforward.
The config is a JSON file that contains two things: the settings that you want to run the tests (in this case, we want to run them on Firefox) and if you want to run them on Sauce Labs’ infrastructure. You have to tell Sauce your username and access key before this can happen, but it’s not a good idea to commit your Sauce access key to a GitHub repository.
So instead, we’re going to put the username and access key into environment variables. As long as we run the interpreter with the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables set to the right values, it will work fine. And how we set those variables, we’ll get to in a moment, as that will be in the Travis configuration.
Here we have the tests we actually want to run. This config format supports glob syntax, so you don’t have to list each test individually. Just put in the path to the directory with an asterisk at the end, and it will find the tests. And it will ignore anything that’s not a JSON file.
That’s the entirety of the config, so let’s save that in our repo.
And commit and push it.
Now the next thing we need to do is to tell Travis that we exist, and that we would like to use its services.
You can log yourself in…
…using your GitHub account.
And then you go in here…
…and you enable Travis integration for this particular repo.
OK. Now the final step is to configure Travis. We need to tell Travis what running an integration test for this particular project actually means. That’s done by putting a file into the top level of the repository, which tells Travis what to do. It’s called .travis.yml.
So again, I’ve just pasted in something.
Now we tell it to install the Selenium interpreter, and then run it with the configuration we already made. And the one thing we’re lacking here is the environment variable for the Sauce access key. Setting variables in Travis is easy if you don’t mind them being public, which you probably don’t mind for your Sauce username.
Now this is the one tricky bit in this tutorial. You need to encrypt your access key to let Travis know what it is without revealing it to the world. The way to do that is to first install a Ruby gem.
Once that’s installed, make sure you’re in the repository you want to encrypt something for and log in to Travis.
Then you can use this command to encrypt an environment variable.
What comes out is a blob of encrypted data…
…which you can paste into the .travis.yml config file like this. The encrypted data is tied to your git credentials and this particular repo, so it should be safe from snooping or theft. Your access key should also be safe. If it does ever become compromised, you can also generate a new one. That’s what I’ll be doing in a few minutes.
Now you can save this file…
…commit it, and push it. And of course this means Travis now goes “Oh, this has been pushed to! OK, is there a .travis.yml? Yeah, there is.”
Let’s actually go onto Travis. We can see it’s in the queue right now.
Ok, good, it’s running the tests.
You can see the details of each test being run.
But you don’t need to know any of the details, because in the end, we get an email saying, “Hey! It’s succeeded, everything’s fine!”
So I’ll just show you what happens if you break things. You can edit these JSON scripts with a text editor, which is pretty easy.
Let’s go in here and change this verify to something that’s not actually the case.
Save, commit, and push, and Travis runs again.
And now we get told that the build is broken! One of our tests failed!
So let’s fix the test up again.
Save, commit, push.
And shortly after, Travis tells us that we’ve fixed the build. Excellent.
That’s all really – you can use these four components to make a continuous integration system for your site. If you have any questions, drop Adam or me an email.
Since the first release of Selenium Builder two years ago, we’ve been focusing on extending its capabilities and improving the underlying code. For those not familiar, Selenium Builder is an open source tool that helps you easily build and export Selenium tests using record and playback capabilities, and then export those scripts to any programming language.
After many months in the wild and tons of great feedback from the community (plus pull requests and patches on Github), we are incredibly happy to announce the release of Selenium Builder v.2.0 and an updated version of Sauce for Selenium Builder.
Watch the video below to see what’s new and read on for more details.
Selenium 2/WebDriver Support
The most important feature to come along in Selenium Builder is built-in support for Selenium 2/Webdriver. You can record, edit, and play back test scripts based on the new Selenium version, which is particularly handy for folks needing help making the switch from Selenium RC to WebDriver. Plus, since Builder uses the same interface for this as it does for Selenium 1, you can use a mixture of both technologies to build your scripts. Selenium Builder can also translate between the two versions with reasonable accuracy.
Additionally, you can export Selenium 2 scripts into a wide variety of languages: Java, Ruby, Python, PHP, Node.js, and C# (with more language support to follow). Se Builder 2 also defines a clean JSON-based format for storing scripts for later editing. These JSON-based scripts can be played back by a standalone interpreter, so you don’t have to commit to a particular programming language for playback.
There’s another exciting addition to Selenium Builder and that’s support for plugins that extend Builder’s functionality. Plugins can be installed and managed directly from the user interface of Builder. The two plugins listed below are available in this release, with more on the way!
Sauce for Selenium Builder Integration
With this plugin, you can run scripts on Sauce Labs directly from Builder’s interface. Simply export your tests to the programming language of choice and pick from the 100+ browser/OS combos we support to run against. To download Selenium Builder with the Sauce capabilities already baked in, click here.
This plugin lets you store your scripts on GitHub for safe storage and easy collaboration.
Foreign Language Support
And last, but certainly not least, Selenium Builder is now available in German and French (as well as English).
If you’d like to improve Selenium Builder by adding new languages, creating a plugin, or just fixing issues, have a look at the documentation and drop us a line!
After a considerable period of investigation and bug fixing in collaboration with the Selenium project and the Legion of the Bouncy Castle, we are happy to announce we’re moving to Selenium 2.30.0 as the default version in our service. You can find more details about this release in the official changelog.
This transition is scheduled for Friday, March 1st. In the meantime, we advise you to try your tests on this new version by adding the following key-value to your tests’ Desired Capabilities:
If you see any issues after moving over to this new version, we definitely want to hear about it. And remember, once we move everyone to 2.30.0, you can still keep your tests on our previous default (Selenium 2.18.0) version by using the “selenium-version” capability outlined above.
Today we are extremely excited to announce the initial release of Appium on Sauce, a new way to automatically test your native and mobile web hybrid iOS apps in the cloud.
Appium on Sauce has its roots in Appium, an open source project written in Node.js that draws from Dan Cuellar’s work on iOS Auto. Appium currently supports iOS testing with Android support in the works. Using the WebDriver JSON Wire Protocol to drive Apple’s UIAutomation, Appium takes Selenium commands from your app and translates those into a readable format for UIAutomation. With this seamless transition happening under the hood, Appium requires no recompiling or modifications to your app, and allows you to write tests in your favorite programming language and testing frameworks using the Selenium API and language-specific libraries.
Support for mobile web hybrid and native app testing has been on our radar screen for a while now, particularly since the Mobile Testing Summit, where we were first introduced to the project now known as Appium. Given its out-of-the-box compatibility with Selenium, goal of cross-platform support across iOS and Android, and flexible approach to automation, pairing Appium with our cloud to provide a turnkey service for automatically testing native and hybrid apps made perfect sense.
And the best part? Appium on Sauce requires no setup or maintenance of iOS infrastructure. Tests can run in parallel across dozens of machines, allowing them to complete much faster than if they were running serially. Additionally, you can plug Appium on Sauce into your existing CI setup and let the cloud serve as your build server.
To get a quick of idea of how Appium on Sauce works, check out the demo video below featuring Everest, an iPhone app that helps you achieve personal goals.
For the next few weeks, Appium on Sauce will be available by invitation-only as we grow our ability to support more users. To sign up for an invite, visit http://saucelabs.com/appium and tell us a bit about your app and why you’d like to test it in the cloud. We expect to open up Appium on Sauce to everyone in the coming weeks so stay tuned to our blog and Twitter for an announcement on that.
And in the meantime, if you’re a Node developer working on a mobile app development or testing, please consider contributing to Appium to help make it better!