Getting the Most Out of Selenium with CloudBees and Sauce Labs

December 18th, 2012 by The Sauce Labs Team

We recently conducted a webinar that illustrated how you can get up and running with CloudBees and Sauce Labs with one click, using the Sauce Clickstarts for CloudBees.

There were a number of questions raised in the webinar that we didn’t get time to address, so we’ve included answers below. If you have any further questions, please include a comment below.

Q: If you define that you want to deploy to RUN@CloudBees in pre-build step , is it guaranteed that the Maven job will be run only and only if deployment is finished?
A: Yes, that’s correct.

Q: How do you satisfy that application is deployed before Sauce tests are run?
A: The CloudBees Deployer plugin will return the status of the deployment: the next test step won’t run until the deployment is successful.

Q: In the JBoss example, tests are run against an in-memory JBoss. Is it possible to run the tests against the RUN@CloudBees instance?
A: Yes, both configs are possible. In order to run against a different JBoss instance, you can specify your Jenkins build to use a different profile (which corresponds to a separate Arquillian container).

Q: Where is the sauce-jboss-clickstart and sauce-java-clickstart source code located?
A: and  Please feel free to fork!

Q: How are code changes moved to the cloud after changes are made locally?
A: What happens is you create a “cloud” repo with GitHub or CloudBees and then clone it for local development – then push changes back up to the cloud and the Jenkins jobs run automatically.

Q: So SSH is used to move code to the cloud after the build?
A: SSH is the default but there are other options

Q: Can we have two ClickStart applications active at the same time?
A: Yes, you can have multiple applications running at the same time.

Q: So the browser set in Jenkins config creates an environment variable that will override the default Sauce Labs SELENIUM_BROWSER?
A: Yes, the Sauce Jenkins plugin will set a series of environment variables that represent the selected browser(s).  Further information on the environment variables set by the plugin is available from the Jenkins tutorial page.

Q: What are the browser requirements for the Sauce browser test screen to capture video?
A: You don’t need to have a certain browser for that. Your test will automatically be recorded and screenshots captured every time you run a test on Sauce.

Q: “Drone extension for arquillian” huh, what? :-)
A: I know, right? Take a look at our recent blog post for more information. :-)

Q: I read that it is possible to test applications deployed behind firewalls, can you explain how this works?
A: Yes, you can use Sauce Connect to test websites running on a local webserver.  The Sauce Jenkins plugin can be configured to launch Sauce Connect prior to the running of your testss.

Q: Is it possible to select more than one browser/OS combination in the same job?
A: Yes, it is possible to set more than one browser – in this case, the Sauce plugin will set a SAUCE_ONDEMAND_BROWSERS envionment variable that contains the browser information (in JSON format).

Q: Do users of Sauce need to log in via the CloudBees environment first? Do users have to have a CloudBees account in order to use the Sauce service?
A: No, if you just wanted to use Sauce, you could sign up via If you want to use it with Cloudbees, however, we recommend signing up through Cloudbees.

Q: What needs to be configured on the Jenkins build job to allow the Sauce test run links to appear in the Jenkins build job?
A: You will need to output the information to the  stdout in the following format:

SauceOnDemandSesssionID=<session id> job-name=<some job name>

where ‘session id’ is the Selenium session id and ‘some job name’ is an identifier for the test under execution (usually the test name).

When the job has completed, the Sauce Jenkins plugin will parse the build output, and if it finds the above output, will update the Sauce Job to store the Jenkins build number.

Q: Is the Sauce plugin built to support Node.js/Ruby/.NET/PHP apps?
A: Yes! The Sauce plugin for Jenkins supports all languages.  In the coming weeks, we will add more Clickstart examples that demonstrate some of the other languages and platforms supported by CloudBees.

In Case You Missed It: Mobile Testing Summit Recap

November 29th, 2012 by Ashley Wilson

One month ago, we organized and hosted the Mobile Testing Summit, a one day conference devoted to all things about mobile test automation. In reflecting on this event (which, I must say, went swimmingly well), I thought I’d pull back the curtain and give you a look at how it came to be and what we all got out of it.

The idea for the MTS started with Jason Huggins, Sauce’s cofounder and CTO, in July 2011. Jason is our idea guy and he’s always coming up with new (and, at times, crazy) things to work on, whether it’s pushing for the development of Sauce TV, a feature that lets you watch live video of your test running in the Sauce cloud or building Appibot (née BitBeambot), a Selenium testing robot that sits on your desk and tests your mobile devices (and plays Angry Birds when off duty).

When he shared his vision for the Summit that day in July, I enthusiastically told him we should make it happen. Having recently organized the inaugural Selenium Conference, I was ready to get my hands dirty with another big-scale dev event. And since mobile testing was especially new that summer, the topic felt ripe for the picking. We spent that afternoon bouncing thoughts around and getting pumped to pull it off.

And then…nothing happened. As much excitement as that initial conversation garnered, I think we both started to question whether the timing was right. After all, Sauce didn’t have anything related to mobile testing in our service at the time and there were only a handful of related open source projects, many of which were in their infancy.

But the Mobile Testing Summit proved to be the little idea that could, and it stayed in the back of our heads until the following spring when Dante Briones (who later came on as co-organizer) got wind of it and encouraged us to go forward. By then, the Saucers were all hands on deck putting Mac & mobile into the cloud for our customers, and projects such as Frank, Calabash, and Firefox OS were getting considerable attention. Plus, we knew there was interest by the number of people writing in to the Sauce’s help desk asking for mobile. So, this time, we did.

The vision was simple. Get all the awesome developers working on open source mobile test automation tools together in one room to talk. We wanted to hear about the key challenges everyone faced and why he or she chose a certain approach. And we wanted to come up with a roadmap for mobile test automation and see if there was cohesion amongst projects. After deciding on the size of the Summit (80 people seemed about right to foster meaningful conversations), we started asking around to see who might like to participate.

As it turned out, folks were ready for an event just like this. It wasn’t long before we had confirmation from all the speakers we asked, and had sold out. With sponsorship from the likes of Facebook, eBay and others, we decided to open up another 50 tickets and those went quickly too. Which meant one thing…the Mobile Testing Summit was actually happening. And though it veered a bit from our original vision, what resulted this past November was even better than what we imagined.

Not only did attendees get an introduction to some of the best tools for testing an iOS or Android app, they also got an inside look at how companies such as Facebook test their heavily used hybrid mobile app. They got to see how Mozilla is handling test automation for its new Firefox OS and how Sikuli, an open source project that uses image recognition to identify and control GUI components, can be used to test with mobile. Attendees got to spend the day chatting with kindred spirits in the mobile automation space, and speakers got to liaison with other devs pioneering this niche market. Projects were merged, feedback was shared, and a mailing list was started to keep the conversation going well after the Summit ended.

Basically, the start of a new community was born.

Given the success of this first event, you can guess we’ve already got our sights set on another Mobile Testing Summit next summer. We will keep some things the same, such as putting together a varied mix of speakers, catering from a local food favorite like Bi-Rite (which was de-LISH), and picking a venue that is equally artsy and conference atypical as the Terra Gallery was. And we are thinking about how we can change some things up too, like perhaps extending the event over two days, putting out a call for proposals while also inviting certain speakers, adding a workshop / hack session and building in even more time for conversing and idea sharing.

We are excited to see where the community will be in 9 months time (no doubt a lot will change) and we hope you’ll join us for the second Mobile Testing Summit. If you’re curious to know what was covered at this year’s event, be sure to check out the videosslides, and Jason’s keynote address. Happy (mobile) testing!

 The Sauce crew at the MTS after party

SF Selenium Meetup: Proxies, HAR Files, JS Executors & Testing Flex/Flash

July 31st, 2012 by Ashley Wilson

Noted Selenium contributor Adam Goucher paid a visit to the SF Selenium Meetup group on July 19th to talk about getting the most out of WebDriver. From proxies and HAR Files to JS Executors and testing Flex & Flash testing, he gave us plenty of great takeaways for the newbie and experienced tester. See for yourself!

#SFSE Video: Stripping Down RemoteWebdriver

March 13th, 2012 by Ashley Wilson

For our February San Francisco Selenium Meetup, Santiago Suarez Ordoñez, Sauce Ninja and Selenium Contributor, dove in to the RemoteWebdriver codebase and emerged with a highly technical talk that covered everything from the DesiredCapabilites object to binding implementations and caveats.

For those unfamiliar, RemoteWebdriver lets you run your Webdriver tests remotely and use Sauce Labs or Selenium Grid to scale your testing. In Santi’s opinion, it’s the best driver out there, and if you have a look at the video below, you’ll get a sense as to why that is. Watch along as he gives a brief overview of what RemoteWebdriver is before going into its architecture design, the JSONWireProtocol, and the pros and cons for using it over other drivers.

Thanks to our friends at Huddler Inc for hosting AND providing ample pizza and beer to keep us happy for the night. To learn more about the San Francisco Selenium Meetup group and attend a future event (we’ve got one tomorrow night!) visit our meetup page. And if you’re interested in presenting at or hosting a meetup, please get in touch!

Top Selenium Tips From The Sauce Codebase

October 13th, 2011 by Ashley Wilson

Having run more than 8 million tests in the Sauce cloud, we’ve learned a thing or two about the common pitfalls folks encounter when writing and running Selenium tests.

To help others not make those same mistakes, we recently started hosting bi-weekly webinars led by Sauce Ninja Santiago Suarez Ordoñez. This week’s webinar covered various Selenium tips, including implicit waits, timeouts, and the reasons to avoid complex locators.

If you’d like to attend an upcoming webinar, register here. Happy testing!

How To Test Behind Your Firewall With Sauce Connect

September 14th, 2011 by Ashley Wilson

Check out the video from yesterday’s Firewalls and Testing in the Cloud webinar to see how to connect your local and firewalled servers to our cloud using Sauce Connect.

We give live tutorial webinars every two weeks. If you’d like to attend an upcoming session, visit our webinars page.

Happy testing!

Tips From Our Codebase To Help You Write Reliable Selenium Tests

July 26th, 2011 by Santiago Suarez Ordoñez

As part of our new “Test Like A Ninja Webinar Series,” I held an improv webinar last week that covered how to write better Selenium tests. Most of the content came from my own Selenium experience, as well as experience gleaned from working on our own customers’ issues.

Without further ado, here’s the video:

And here are some of the code snippets I talked about:

Implicit waits in Selenium 1:

Ignoring Open and waitForPageToLoad failures, as well as reporting pass/fail status automatically:

Reporting pass/fail status automatically on python for Selenium 2 tests

Tune in to the next Test Like A Ninja webinar on August 2 to learn how easy it is to test behind a firewall in the cloud.

Happy testing!

Javascript + Selenium: The Rockstar Combination of Testing

July 25th, 2011 by Ashley Wilson

For our July Selenium meetup, held last Thursday, we wanted to give attendees something a little different to chew on. Thanks to our good friends at Yammer, who co-hosted the event with us, we did so not only with delicious catered Mexican food, but also plenty of Javascript & Selenium testing goodness to go around.

Bob Remeika, senior engineer at Yammer, gave a spirited presentation that left no one questioning his stance on testing (his opening slide – “Test your shit” – really said it all). He gave us an inside look at how Yammer tests using a combination of Jellyfish and Sauce OnDemand, and gave some great advice on knowing what and how to test when you’re just starting out.


We also had Adam Christian, Sauce Labs’ Javascript Aficionado and the creator of Jellyfish, give two talks. The first, a lightning talk titled “Javascript Via Selenium: The Good, The Bad, The Obvious”, covered some of the lesser known things about Javascript testing via Selenium.

The second showed off how you can use Jellyfish, the open source Javascript runner that he announced a few weeks ago, to run your JS unit tests in any environment.

Thanks to Adam, Bob, and Yammer for making this quite the fun and memorable meetup. As always, the San Francisco Selenium Meetup group is free to join & we meet monthly at different venues around the Bay Area to talk all things testing. See you in August!

Why BeachMint Can’t Live Without Sauce Labs

May 19th, 2011 by Ashley Wilson

When it came time for popular eCommerce company Beachmint to launch their first consumer brand JewelMint last year, they knew automated testing would be a key component in their development process. But they weren’t keen on taking on the overhead costs of setting up, running, and maintaining their own testing infrastructure. So they turned to Sauce Labs.

Watch the video below to see how Sauce OnDemand helps the BeachMint development and QA teams consistently ship quality code while finding and fixing bugs quickly and easily using screenshots and videos of every test. As William Belk, Director of Product Development, puts it, “Without the Selenium/Sauce Labs combination, our life would generally suck.”

Why CSS Locators are the way to go vs XPath

May 17th, 2011 by Ashley Wilson

Last week, our own Santiago Suarez Ordoñez gave a presentation to the San Francisco Selenium Meetup group that convinced us all to say no (for the most part) to XPath and yes to CSS Locators in our Selenium testing.

In his role as official Sauce Ninja and as a prolific poster in the Selenium forums, Santi has helped more users solve locator issues than possibly anyone else in the world. He’s previously written a number of blog posts on ways to improve locator performance. As Sauce CEO John Dunham puts it, “If there was a foursquare mayorship for locators, Santi would have it for a lifetime.”

Drawing from this experience, he gave us these four reasons for using CSS Locators:

1. They’re faster
2. They’re more readable
3. CSS is jQuery’s locating strategy
4. No one else uses XPATH anyways!

I can’t speak for everyone, but Santi sure sold me on point number one when he showed off the performance metric script he wrote a script that tested the speed of XPath vs the speed of CSS Locators. There wasn’t much of a difference in Firefox, Safari, or Chrome, but with IE, the results were undeniable. Take a look:

To underscore this even further, he also recorded a video in Sauce OnDemand that uses one heck of a cute kitten to illustrate just how slow XPath can be. The cat’s paw movements represent the test clicking through the different locators. The first batch of clicks uses CSS Locators and completes in under 30 seconds. The second batch, the XPath one, continues on for another eight minutes. Eight minutes!

During the rest of the presentation, Santi dives into writing both basic and more advanced CSS Locators. He also spends some time talking about when you shouldn’t use CSS Locators (yes, there are a few cases where it is not the right tool for the job). To see the talk in its entirety, check out the recording below. And if you’re thinking of switching over from XPath, but unsure of how to go about it, check out the nifty tool Santi wrote called cssify. It does the handy work of translating your XPaths to CSS automatically.

Helpful links
View Santi’s Presentation Slides
Follow him on Twitter
Follow Sauce on Twitter
Join the San Francisco Selenium Meetup group!