Online Workshop – Testing Behind a Firewall

March 6th, 2014 by Amber Kaplan

Learn all about best practices and strategies for testing apps behind a firewall and how to use Sauce Connect during our next online workshop! Join speaker Mike Redman, Director of Sales Engineering at Sauce, on Tuesday, March 11, 2014, at 11:00 AM Pacific Time for the latest.

Whether you’re at the enterprise or startup level, security is a hot topic. That’s why we created Sauce Connect. Sauce Connect creates a secure tunnel between your firewalled app and the Sauce cloud so you can run your tests knowing that your data is encrypted through industry standard TLS.

Keeping our security standards in mind, we completely rewrote the app. With the launch of our latest version, Sauce Connect 4, your tests will now run faster than ever, even with heavy loads. It’s better performing, more reliable, and gives broader support for a wider range of web standards, including Websockets.

Mike will walk you through testing behind a firewall and how to use Sauce Connect 4.  A live Q&A session will follow.  Register today!

Meet the New Sauce Connect 4

February 26th, 2014 by lauren nguyen

Sauce Connect 4We know a thing or two about testing securely, and we know that testing behind a firewall is crucial for many of our users. That’s why we’re excited to announce that we just released Sauce Connect 4, a brand spanking new version of our secure tunneling app.

Sauce Connect creates a secure tunnel between your firewalled app and the Sauce cloud, enabling you to run your tests on Sauce while keeping your data secured. Data transmitted with Sauce Connect is encrypted through industry-standard TLS. So what’s new with v4?

To start off, we rewrote the entire app from the ground up to make it simpler and better performing, especially under high loads. That means your tests run faster than ever, even if you’re running large numbers of tests. And it’s more reliable than ever. The rewrite gives SC broader support for a wider range of web standards, including Websockets.

Sauce Connect 4 also supports working entirely through a proxy, if that’s what you’re into.  And of course we squashed some bugs.

Sauce Connect secure tunneling architecture

We’ll be holding an online workshop on Tuesday Mar. 11th, 11AM PST about testing behind a firewall, where we’ll cover best practices and strategies for testing apps behind a firewall and discuss everything you need to know about Sauce Connect. Register here.

Want to get started using Sauce Connect or learn more? Check out our Sauce Connect docs page.

TeamCity Plugin Now Available for Sauce

January 23rd, 2014 by lauren nguyen

teamcity sauce integration CIWe just finished a plugin that allows you to integrate your Sauce tests with JetBrains TeamCity! Hooray! We hope the new integration will help anyone using TeamCity for CI use Sauce to test all the things. The integration will allow you to run your tests through TeamCity and view the test results, including steps performed, screencast, and screenshots, inside TeamCity. You can get the integration set up here.

A Word About Our Growing Pains

November 7th, 2013 by Adam Christian

Over the last year, the Sauce customer base has been growing tremendously. We have heard your concern about our recent service interruptions. We have always had our sights set on 99.99% uptime, and are continually doing everything we can to achieve that. Examples include our recent datacenter move, constant capacity upgrades, and complete re-architecture of our Mac cloud.

Unfortunately, our growth has caused some hiccups that have negatively impacted your experience using Sauce. Knowing ourselves how important a smooth development process is to productivity, our recent quality of service is simply unacceptable and we vow to do better. We want it to be 100% clear that reliability is our top priority, and every time we can’t run jobs within single digit seconds, it becomes a red alert of the highest priority for our team.

To give you insight into how we are assessing situations such as the one yesterday: when our average launch time for a job exceeds 30 seconds, we consider that event an outage and launch an investigation to fully understand what happened, and why, with the onus on not only fixing the problem, but improving our process going forward. These include such things as eliminating single points of failure, scaling our cloud capacity far beyond the foreseen needs of our users, and introducing more stringent testing and analysis into the development and pre-deploy processes.

In terms of communication, when an outage happens we update @sauceops immediately and also status.saucelabs.com, which shows the average wait time for jobs. Our status page is continuously monitoring the situation and updating every few seconds. We also take care to update @sauceops when we notice that job wait times are exceeding an average of 10 seconds, though we don’t count that as an outage.

Additionally, we’d like to make a suggestion to help isolate your build from the effects of unexpected elevated wait times and internet latency should they happen in the future. If you are using a CI system or test runner that times out and fails tests, please consider increasing your time-out threshold to five minutes to isolate your suite for when the Sauce Labs cloud experiences longer wait times.  This will allow your jobs to remain queued rather than fail on Sauce. Again, you can go to http://status.saucelabs.com at any time to see average wait times and tweets from @sauceops.

Rest assured that we take events like yesterday extremely seriously, and are working to prevent this from happening in the future. We are fortunate to have very loyal customers and don’t want to let you down.

- Adam Christian
VP of Engineering
Sauce Labs

Announcing iOS 7 support in the Sauce Labs Cloud

October 31st, 2013 by Ashley Wilson

ios7_01

Today we’re excited to let you know that you can now test your hybrid and native apps with iOS 7 on Sauce. With more than 200M downloads since its release, we know that this is an important platform for our users to test on.

To get started, visit our browsers and platforms page for Appium and copy and paste the DesiredCapabilities provided for either version of iOS (or Android) you want to test on. And if you’re new to mobile test automation in general, check out our Getting Started with Appium guide.

If you have any issues, give us a shout. Otherwise, happy (iOS 7) testing!

Error Messages on Sauce and What They Mean

October 21st, 2013 by lauren nguyen

Ever run into an error message on Sauce you don’t understand? Alex put this handy list of error messages on Sauce together. You can find the original doc here. This will soon be available for reference on Sauce’s website, but in the meantime, brush up on the error messages you might see on Sauce and what they mean.

==== AUTOMATED JOB ERRORS: ====

– Invalid credentials –

Some combination of the following error messages will be thrown:
OpenQA.Selenium.WebDriverException : Unexpected error. Unknown username.
You sent username ‘None’ in your browser string, which is not a valid Sauce Labs account.
OpenQA.Selenium.WebDriverException : Unexpected error. Invalid Credentials.
org.openqa.selenium.UnsupportedCommandException: Invalid Credentials.
You sent the access key ‘None’ but it does not match the access key associated with your account. Please login to saucelabs.com to retrieve a valid access key.
This indicates that Selenium could not parse the credentials you sent. If you’re using “access-key” in your test setup, try replacing it with “accessKey”, or vice versa, as it’s language-dependent.

– User Terminated –

This means that you ended the job, either through the “Cancel Job” button, by jumping into an automated session from the Sauce website, or via a breakpoint. Since this terminates the job immediately and hands over control to you, the test will not upload any screenshots, video, or logs that were collected previously.

– Timeout errors –

1. “Test did not see a new command for 90 seconds. Timing out.”

2. “Selenium took too long to run your command”

3. “Test exceeded maximum duration after 1800 seconds”

Sauce builds in a few default timeouts to keep uncontrolled tests from eating up your minutes. Here’s a brief rundown on what each of the available timeout settings does:

1. The “idle-timeout” (default 90 seconds) ends the job if no commands are received from your Selenium script.

2. The “command-timeout” (default 5 minutes) ends the job if a single command takes too long to run.

3. And the “max-duration” timeout (default 30 minutes) ends the job if your entire test takes longer than you want.

You can set these to different values using “desired capabilities” settings in the setup of your test, as described at the URL above.

– Unsupported OS/browser/version combo –

Check to make sure that the browser, browser-version, and platform settings you’re using are in our supported list.
And note that there’s a separate list for Appium tests (as they use a different mechanic to drive the tests).

– Browser failed to start –

The twin sibling of “Unsupported version”, this message means that something a little more unusual is off in your test setup. Usually, this means that you’re specifying a Selenium version that isn’t compatible with the browser/version/OS you’ve selected. (For example, you should not be setting this for any mobile tests.)

Try simply omitting that setting, and if you still see the issue, feel free to write to help@saucelabs.com with a description of the issue and a copy of your setup code.

– Client disconnected during getNewBrowserSession request –

This means that your test runner decided to end the job before it had fully initialized on Sauce’s end. There are a few potential causes:

1. You’re running too many tests at a time: Check the left sidebar on your Account page. It shows a “Parallel tests” number, which is the maximum number of tests you can run at a time, based on your subscription level. If your account can run 2 parallel tests, and you’re launching 10, 8 will be “queued” until one of your tests finishes and a slot frees up. However, if this takes a long time, your test runner may choose to end the queued jobs after a few minutes instead of waiting. Just make sure you’re launching a reasonable number of simultaneous tests for your account.

2. High job wait times: Check our status page and/or follow @sauceops on Twitter for up-to-the-minute news about any issues within the service. If something causes demand for certain VMs to stack up, your jobs may be queued and (as above) terminated by your test runner.

Tests that end this way are never taken out of your minutes.

– Runaway execution. Please contact help@saucelabs.com for assistance. –

This message means that an error has been detected in the VM or in Selenium, which caused the test to behave abnormally, and Sauce detected this and shut down the job. These are very rare and usually do not recur. If you do see more than one on the same test, let us know.

==== MANUAL JOB ERRORS: ====

– Job does not load –

There are two common scenarios here:
• Error message: “Uh oh! Some error occurred while connecting to the browser”
• The job seems to start, but you see only a white text box in the middle of a black screen.
Both of these indicate that your browser is having trouble displaying the VNC stream from the remote machine. Take the following steps to troubleshoot:
• Check the video on Sauce: If the recorded video after the job shows a steady video stream, this indicates that the issue is in your computer or connection to Sauce. However, if the Sauce video shows the same issue, that indicates an issue in our service. In that case, send us the URL for the job page and a screenshot of the issue.
• Check that your browser is up to date: If you’re on an older version, this may cause incompatibilities. Update your browser and try again.
• Check your firewall: make sure that your machine allows full access for the interactive stream over the required ports.
• Check that the Internet connection is stable: We recommend running Sauce tests from a machine with a wired Ethernet connection, to ensure a steady connection. If the connection flickers, this error could be thrown.

==== FRAMEWORK-SPECIFIC ERRORS: ====

– Appium: Failed to download mobile app –

First, if you’re having trouble, try using Sauce’s temporary storage. Uploading your app to our servers makes it download faster.
If you’re using this, make sure that you’re specifying the “APP_PATH” as “sauce-storage:your_file_name” — NOT the saucelabs.com/rest/v1/storage… URL.
And when you upload your file to our server, check the JSON response. It should be formatted like this:
{“username”: “USERNAME”, “size”: 99999, “md5″: “bc01aad123456789glhf9ee6e8c7eab3″, “filename”: “AppName.zip”}
Make sure that the size is something reasonable for the .zip file you uploaded. If not, check your file and try again. This should resolve the issue.

Re-introducing the Sauce Launcher Plugin for Google Chrome

April 24th, 2013 by lauren nguyen

For all of you who run manual test sessions with Sauce Labs, we’ve got some great news. We recently released our updated Sauce Launcher plugin for Google Chrome, with lots of improvements to help you run manual jobs more easily.

Sauce Launcher for Google Chrome

Sauce Launcher makes launching a manual test as easy as clicking a button, and we think the update makes it even better. We updated the plugin’s look, feel, and layout to better match saucelabs.com, in order to make it easier to use. Sauce Launcher now also offers all the browsers and platforms we support, and it now automatically updates when we add newly supported platforms and features so it will never go out of date again! Download it in the Chrome webstore, and let us know what you think. Happy testing!

Announcing Support for Lynx in the Sauce Labs Cloud

April 1st, 2013 by Steven Hazel

Today we are extremely excited to announce the initial release of an important browser platform on Sauce: the classic — but innovative — text-based browser Lynx. While Lynx is not as popular as more modern, graphical browsers like Internet Explorer 6, it has a devoted following and a compelling feature set, and we feel it’s important that Sauce customers be able to test their web apps in a text-based browser.Screen Shot 2013-03-31 at 9.23.05 PM

Ever since Twitter demonstrated the power of simple, textual interfaces, we have been looking at Lynx as a bold way forward for the web. We think its less-is-more approach represents the future of sustainable web technologies. Because it spends less energy on graphics, Lynx probably has the smallest carbon footprint of any web browser out there, and lightning-fast performance.

Lynx also has advantages for developers: its javascript support is easier to understand and has fewer pitfalls than many modern browsers, and its support for key web technologies like Flash and Java Applets was decades ahead of its time, and even today is on par with the latest browsers for iOS and Android.

And because Lynx is so different from other browsers, we believe it’s extra important that developers be able to test their web apps against it. That’s why we’re excited to extend Sauce Labs’ cross-browser functional testing cloud to include interactive and Selenium testing for Lynx.

Give it a try today. Happy testing!

Share Your Live Sauce Manual Sessions for Real-Time Debugging

March 27th, 2013 by Bernard Kobos

We recently added a new feature to our manual browser testing web client that we’re excited about. The “Live Share” feature in Manual Sauce, our interactive testing tool, allows you to share your live browser sessions with coworkers simply by sending a link. You can collaborate using your favorite IM client to demonstrate the exact steps to reproduce a bug you’ve run into and interact with the browser running on Sauce at the same time!

live-share-button

To try this out, first start a manual testing session. Once your browser has loaded, look to the header bar for the live share button next to the URL bar. Hover over the button to pop open a dialogue box that lets you specify the email address of the person you want to receive a private, unique link for sharing the manual test session. Then click the share button.

live-share-dialog-url

If you leave the email boxes empty and click the share button, no email will be sent but your test’s visibility will change to ‘share’. This means anyone with the link can access the test session and view it live. If it’s easier to share the session directly via Gmail or whatever email client you prefer, just copy the link from the live sharing dialog and send it to your coworker(s).

live-share-enabled

To indicate that the tests can by accessed by anyone with proper link, the sharing button will be turned on next to the address bar. You can click this button to turn off live sharing mode. Each time you click that button, it makes a call to our REST API and the test visibility mode is updated (http://saucelabs.com/docs/rest).

Please note that by default, the job visibility mode is set to ‘private’. This way, only the owner of the test can access it. When you click the button, the mode is set to ‘share’, allowing everyone with valid secret link to access the test page.

We hope you find this new feature useful. And as usual, let us know if you have any questions. Happy testing!

Building a CI System Using Selenium Builder, GitHub, Travis and Sauce Labs

March 11th, 2013 by David Stark

Today I’m going to show you how you can use Selenium Builder with GitHubTravis, and Sauce Labs to make a continuous integration solution for your website that’s all on the Internet, nothing stored locally. You can either follow along with the demo video below or read on for the written tutorial.

The great thing about this setup is that if you put your site code and your tests into the same GitHub repository, then whenever you update the site code, and whenever you update the tests, Travis will rerun your Selenium tests on Sauce OnDemand – and then send you an email about whether or not they still work!
Screen shot 2013-03-07 at 14.11.55

So you can get a full Selenium CI solution by putting together these components. And because they’re all components, if you want to use something else in part – for example Jenkins instead of Travis – you can construct your own solution.

For now I’m going to show you this set of four things that work really well together. The great thing about this is that you store everything on GitHub, so all your tests are stored in the cloud, nothing is stored on some local machine somewhere. You can record and edit tests from any computer that runs Firefox. You can access your tests, edit them, & play them back locally. And all of the actual execution of the testing – the CI – is handled by Travis and Sauce. So you actually don’t need to build any of your own testing infrastructure!

OK! So how do you actually do this?

The first thing you need to do is install the plugins. You go into Selenium Builder, you go into Plugins, and you just click install on GitHub integration and Sauce. Restart Builder.

Screen shot 2013-03-07 at 14.12.19

Next up, you want to record some tests. Let’s just say for now we’re testing this website.

Screen shot 2013-03-07 at 14.12.34

Let’s record a verification.

Screen shot 2013-03-07 at 14.12.39

Now we’re going to save this test. In the File menu, save the test to GitHub.

Screen shot 2013-03-07 at 14.12.44

Because this is the first time we’re using it, I’m going to have to enter my details.

Screen shot 2013-03-07 at 14.12.50

Now, as you can see, we can access our repositories. And I’ve set up this little repo here as an example one.

Screen shot 2013-03-07 at 14.12.53

So let’s save this test in here, as JSON, which is the standard format Builder works in. This means we can later open it again and edit it.

Screen shot 2013-03-07 at 14.13.05

Let’s do another test.

Screen shot 2013-03-07 at 14.13.18

Now we save that one as well. Just record as many tests as you want and put them all into this place.

Screen shot 2013-03-07 at 14.13.31

Now we’ve done that, the next thing we need to do is configure the Selenium interpreter, which is actually a little node.js program that takes these JSON scripts and uses Webdriver to play them back.

Let’s clone the repo to disk so we can add the config files we need.

Screen shot 2013-03-07 at 14.13.55

Screen shot 2013-03-07 at 14.14.06

Create a new text file. I’ve just pasted in a bunch of things, which are actually reasonably straightforward.

Screen shot 2013-03-07 at 14.14.15

The config is a JSON file that contains two things: the settings that you want to run the tests (in this case, we want to run them on Firefox) and if you want to run them on Sauce Labs’ infrastructure. You have to tell Sauce your username and access key before this can happen, but it’s not a good idea to commit your Sauce access key to a GitHub repository.

So instead, we’re going to put the username and access key into environment variables. As long as we run the interpreter with the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables set to the right values, it will work fine. And how we set those variables, we’ll get to in a moment, as that will be in the Travis configuration.

Here we have the tests we actually want to run. This config format supports glob syntax, so you don’t have to list each test individually. Just put in the path to the directory with an asterisk at the end, and it will find the tests. And it will ignore anything that’s not a JSON file.

That’s the entirety of the config, so let’s save that in our repo.

Screen shot 2013-03-07 at 14.19.17

And commit and push it.

Screen shot 2013-03-07 at 14.19.36

Now the next thing we need to do is to tell Travis that we exist, and that we would like to use its services.

Screen shot 2013-03-07 at 14.19.44

You can log yourself in…

Screen shot 2013-03-07 at 14.19.46

…using your GitHub account.

Screen shot 2013-03-07 at 14.19.51

And then you go in here…

Screen shot 2013-03-07 at 14.19.58

…and you enable Travis integration for this particular repo.

Screen shot 2013-03-07 at 14.20.01

Screen shot 2013-03-07 at 14.20.05

OK. Now the final step is to configure Travis. We need to tell Travis what running an integration test for this particular project actually means. That’s done by putting a file into the top level of the repository, which tells Travis what to do. It’s called .travis.yml.

So again, I’ve just pasted in something.

Screen shot 2013-03-07 at 14.20.22

Now we tell it to install the Selenium interpreter, and then run it with the configuration we already made. And the one thing we’re lacking here is the environment variable for the Sauce access key. Setting variables in Travis is easy if you don’t mind them being public, which you probably don’t mind for your Sauce username.

Now this is the one tricky bit in this tutorial. You need to encrypt your access key to let Travis know what it is without revealing it to the world. The way to do that is to first install a Ruby gem.

Screen shot 2013-03-07 at 14.22.58

Once that’s installed, make sure you’re in the repository you want to encrypt something for and log in to Travis.

Screen shot 2013-03-07 at 14.23.17

Then you can use this command to encrypt an environment variable.

Screen shot 2013-03-07 at 14.23.28

What comes out is a blob of encrypted data…

Screen shot 2013-03-07 at 14.23.31

…which you can paste into the .travis.yml config file like this. The encrypted data is tied to your git credentials and this particular repo, so it should be safe from snooping or theft. Your access key should also be safe. If it does ever become compromised, you can also generate a new one. That’s what I’ll be doing in a few minutes.

Screen shot 2013-03-07 at 14.23.44

Now you can save this file…

Screen shot 2013-03-07 at 14.24.02

…commit it, and push it. And of course this means Travis now goes “Oh, this has been pushed to! OK, is there a .travis.yml? Yeah, there is.”

Screen shot 2013-03-07 at 14.24.22

Let’s actually go onto Travis. We can see it’s in the queue right now.

Screen shot 2013-03-07 at 14.24.31

Ok, good, it’s running the tests.

Screen shot 2013-03-07 at 14.24.35

You can see the details of each test being run.

Screen shot 2013-03-07 at 14.24.48

But you don’t need to know any of the details, because in the end, we get an email saying, “Hey! It’s succeeded, everything’s fine!”

Screen shot 2013-03-07 at 14.24.58

So I’ll just show you what happens if you break things. You can edit these JSON scripts with a text editor, which is pretty easy.

Screen shot 2013-03-07 at 14.25.11

Let’s go in here and change this verify to something that’s not actually the case.

Screen shot 2013-03-07 at 14.25.20

Save, commit, and push, and Travis runs again.

Screen shot 2013-03-07 at 14.25.30

And now we get told that the build is broken! One of our tests failed!

Screen shot 2013-03-07 at 14.25.33

So let’s fix the test up again.

Screen shot 2013-03-07 at 14.25.41

Save, commit, push.

Screen shot 2013-03-07 at 14.25.51

And shortly after, Travis tells us that we’ve fixed the build. Excellent.

Screen shot 2013-03-07 at 14.25.55

That’s all really – you can use these four components to make a continuous integration system for your site. If you have any questions, drop Adam or me an email.
Screen shot 2013-03-07 at 14.25.59

Thanks for following along!