Recap: Sauce Labs at Selenium Conf 2014 [VIDEO]

September 24th, 2014 by Amber Kaplan

We’re still glowing from this year’s Selenium Conf in Bangalore. Santi Ordonez, Ashley Wilson, and Isaac Murchie attended the event and represented Sauce Labs.

The Selenium Conference is a volunteer-run, non-profit event presented by members of the Selenium Community. The goal of the conference is to bring together Selenium developers & enthusiasts from around the world to share ideas, socialize, and work together on advancing the present and future success of the project.

Santi, Ashely, and Isaac shared with us some of their insight into the crowd this year that we in turn wanted to share with you. Of approximately 450 attendees, they estimated around 75% purchased their own ticket, therefore the personal interest level in Selenium was high.  The attendees were extremely technical at this event, even more so than in Boston’s conference. Only 5 people confessed to being IDE users, while everyone said they use WebDriver. Win!

Sauce at SelenConf

Santi achieved near star status at the Sauce Labs booth; it was crowded the entire time.


Meanwhile, Isaac gave a great talk titled “Selenium in the Palm of your Hand: Appium and Automated Mobile Testing.”

Image via Selenium Conf 2014

Image via Selenium Conf 2014

Check out Isaac’s talk here:

For more great resources courtesy of Selenium Conf, check out the links below, and make sure to follow Santi and Isaac on Twitter for the latest.

Selenium Conf Resources:

Spread the word: @SeleniumConf

Announcing Sauce Integration With Siesta

September 16th, 2014 by Amber Kaplan

siesta-logoSauce is thrilled to announce that we’ve integrated with Siesta by Bryntum!

Siesta is a JavaScript unit testing tool that can help you test any JavaScript code and also perform testing of the DOM and simulate user interactions. The tool can be used together with any type of JavaScript codebase – jQuery, Ext JS, NodeJS, Dojo, YUI etc. Using the API, you can choose from many types of assertions ranging from simple logical JS object comparisons to verifying that an HTML element is visible in the DOM. It comes in two versions: Lite and Standard. With Lite you can easily test your JavaScript in the browser, and with Standard you can also automate your tests (highly recommended).

Sauce is a cloud-based tool that enables you to securely test your web and mobile apps across 385+ browser/OS/platform/device combinations.

How does Siesta work with Sauce?

Siesta’s integration with Sauce is super easy since they’ve done most of the work under the hood. The main difference from the usual way you run WebDriver tests in Siesta is the addition of a –saucelabs argument, which will define the necessary credentials for connecting to Sauce and the –cap arguments which will specify the OS and browser combinations on which you want your tests to run. When you run Siesta tests using WebDriver, a standard command might look like this:

__SIESTA_DIR__/bin/webdriver http://localhost/myproject/tests/harness.html

For running the same test on Sauce’s hosted browsers, you’ll need to add your Sauce credentials and desired capabilities, like so:

__SIESTA_DIR__/bin/webdriver http://localhost/myproject/tests/harness.html --saucelabs SL_USERNAME,SL_KEY
--cap browserName=firefox --cap platform=windows

This command will then arrange a few things behind the scenes, like setting up the Sauce Connect tunnel so these tests can run locally and securely, and direct Sauce to execute the test specs on the latest version of Firefox on Windows. Of course, there are many desired capabilities and platforms you can choose to make sure you’re running tests according to the coverage that you need, and you can see all the different options on the Sauce Labs Platforms page.

For more information and the nitty gritty details, go ahead and check out the instructions provided in the Siesta docs:!/guide/saucelabs_integration

What does the integration mean?

It’s become more than clear in the last few years that managing all the different browsers and operating systems out there, both desktop and mobile, is a whole lotta hassle and clearly better handled by a cloud-based service like Sauce Labs. But you need to ensure your JavaScript works from everywhere, and since you can now run your Siesta tests on any or all of Sauce’s 385 platform configurations, you can get that assurance with minimal effort and stay focused on the JavaScript code itself without having to invest your time and energy managing infrastructure.

Yep, you read that right: Now you can test your JavaScript code across all of Sauce’s instantly available browsers and platforms without setting up or maintaining your own VMs. Magic!

For more information, check out the Bryntum blog post.

- Michael Sage, Principal Technology Evangelist, Sauce Labs

Michael Sage is a Principal Technology Evangelist at Sauce Labs who helps software teams develop, deliver, and care for great apps. He’s spent over 15 years as a solutions architect and consultant with software companies including Mercury Interactive, Hewlett Packard, and New Relic. He lives in San Francisco, CA.

Recap: How To Combine Front-End and Back-End Testing [WEBINAR]

September 15th, 2014 by Amber Kaplan

sauce_labs_blazemeterThanks to those of you who attended our last webinar, How To Combine Front-End and Back-End Testing, featuring our  Chief Technology Evangelist Michael Sage and BlazeMeter‘s VP of Customer Success, Ophir Prusak. 

Everyone knows that front-end testing is crucial to make sure your web and mobile apps are meeting the needs of your users and customers. But how do you know what will happen to your front end when your web or mobile app is under heavy load?

Michael and Ophir set out to help answer this question, plus more. Together, they covered many valuable topics, including:

  • The fundamentals of approaching performance vs. front-end testing
  • Step-by-step instructions on getting real-world results from your front-end while applying load to the back-end
  • Critical issues you need to know about performance testing

They also showed a real-world test in real time using JMeter and Selenium.

Missed the webinar? You can watch it in its entirety below.

Below you’ll find the top Q&A’s post-presentation:

Q: Why do we need JMeter? Why can’t we do performance testing with Selenium browsers?

A: [Ophir] Great question! The answer is really all to do with scalability. When you’re doing a test with Selenium, you’re usually using real browsers –and real browsers are very resource intensive. On the other hand, when you’re doing a test in JMeter by using virtual users, I can exponentially get a lot more users per machine. Just to give you an idea of ballpark numbers, on a single low-end Amazon EC2 machine, I can get 3,4 or 5 virtual users using browsers in parallel until I hit a bottleneck on the machine. If I’m doing it with a JMeter, I can get 1,000 people in parallel. So you’re looking at around 250X more users when I’m doing a JMeter test per machine than Selenium. So if you’re looking at ten or 100 users, you can do it with Selenium but when you’re looking at tens of thousands of users, it just won’t be able to support it.

Q: Can I test behind the firewall?

A: [Michael] Yeah, absolutely We have a utility called ‘Sauce Connect’, which creates an encrypted tunnel between your environment inside your firewall or your DMZ and our grid and you get a dedicated virtual machine to act as the tunnel property. You run this utility and it appears as if the Sauce Labs grid is inside your network and it’s all done in an encrypted fashion.
[Ophir] At BlazeMeter, we also have the ability to run behind the firewall. It’s a different type of solution. It does require having something we call a BlazeMeter agent, which is basically a load generator which sits behind the firewall, which you can still control through your browser but you have a local machine which is creating the requests for you.

Lastly, please follow our friends at BlazeMeter at @blazemeter, Sauce Labs at @saucelabs, Michel Sage at @mondosage, and Ophir Prusak at @prutwo to keep up with the latest.  Feel free to share this webinar using the hashtag #frontandbackendtesting.



Sharecare Scales Mobile Automated Testing With Sauce Labs [VIDEO]

September 10th, 2014 by Amber Kaplan

We took a whirlwind trip to New Haven, CT to sit down with Daniel Gempesaw, Software Testing Architect at Sharecare. Sharecare is a wellness platform founded in part by Dr. Oz and Oprah.

We learned that after Sharecare started automating their testing process while using Sauce, time spent for each deploy fell from 7 days to 1 day. With more free time, the team is able to focus on scaling their mobile testing with Appium and employing new best practices such as mobile CI.

Watch this video to learn how they scaled their mobile testing with Sauce Labs.

Be sure to check out the case study, too.

Happy testing!

Re-Blog: Add Some Sauce To Your IE Tests

September 4th, 2014 by Amber Kaplan

Sauce Labs hearts ThoughtWorks! And apparently the feeling’s mutual. Check out this great blog post mentioning Sauce Labs by Tom Clement Oketch.

See an excerpt below:

Using Sauce Labs with a Continuous Integration (CI) Service

Running your tests only locally will not get you much mileage, especially if you are working with a sizeable team. Using a Continuous Integration service is therefore essential. Fortunately, Sauce Labs has first class support for a number of Continuous Integration services including JenkinsBambooTravis and TeamCity. The preceding links should contain sufficient information to integrate Sauce Labs with each of those CI services. In our case however, we had already set up Snap-CI as our CI service of choice. Snap-CI currently does not provide such integration with Sauce Labs. We therefore made the following adjustments to include Sauce Labs in our build pipeline:

  • As part of our functional test stage on Snap-CI, it was necessary to set up a tunnel to Sauce Labs using Sauce Connect, otherwise the browsers at Sauce Labs would not be able to run against our application instance in the Snap-CI Build Pipeline. We came across this gist which we altered to suit the requirements of our Snap Build. The gist takes care of downloading, starting and waiting for Sauce Connect to establish a tunnel before the functional tests are actually run
  • The setup remained largely unchanged, except for the use of environment variables rather than the explicit declaration of the Sauce Username and API Access Key. Given that Snap-CI exports a number of additional environment variables during each build, it was also possible to annotate the test descriptions with these variables. Using annotations such as the pipeline counter and the git commit subsequently made it easier to identify the appropriate test in the Sauce Labs test dashboard.

Are you in a position where you need to run IE tests without getting your hands dirty? If so, maybe Sauce Labs can save your time as well. If on the other handyou are more interested in evaluating some of the other options out there, then this guide is a good place to start.

Don’t miss the entire post HERE for more code and instruction.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

How to Combine Backend & Front End Testing [WEBINAR]

August 28th, 2014 by Amber Kaplan

sauce_labs_blazemeterEveryone knows that front end testing is crucial to make sure your web and mobile apps are meeting the needs of your users and customers. But how do you know what will happen to your front end when your web or mobile app is under heavy load?

Sauce Labs and BlazeMeter are teaming up for an awesome webinar, in which we’ll be giving you step-by-step instructions on how to get real-world results from your front end while applying load to the backend.

We will:

  • Reveal the 3 critical things you need to know about performance testing
  • Run a real-world test in real time using JMeter and Selenium
  • Cover the fundamentals of how to approach performance vs. front end testing

Our experts will stay online for a live Q&A session. Plus, at the end of the webinar, we’ll be giving away an exclusive coupon from BlazeMeter & Sauce Labs for a great discount on our solutions.

Sign up today for our event on Tuesday, September 9th, 2014, at  11:00 am Pacific Time.

The Appium Philosophy

August 26th, 2014 by Amber Kaplan

Appium is a rising star in the mobile test automation landscape, and since a few of our developers here at Sauce Labs are regular committers to the project, it’s pretty close to our hearts. You’ll often find Sauce devs hanging out on the Appium Google group answering questions, musing about testing strategies, and helping folks tweak their configurations or hunt down bugs.

When it comes to mobile test automation, in many ways we’re still figuring out what the best approaches are, and with Appium, we’ve had the benefit of lessons learned from early automation solutions that didn’t quite work as well as we’d hoped. Some of these lessons are nicely summarized in Appium’s four-point philosophy:

  1. You shouldn’t have to recompile your app or modify it in any way in order to automate it.
  2. You shouldn’t be locked into a specific language or framework to write and run your tests.
  3. A mobile automation framework shouldn’t reinvent the wheel when it comes to automation APIs.
  4. A mobile automation framework should be open source, in spirit and practice as well as in name!

Let’s explore these four points, shall we?

You shouldn’t have to recompile your app or modify it in any way in order to automate it.

Some early attempts at mobile test automation were based around OCR and pixel-based interactions, which history tells us are never very reliable. After that we saw the introduction of in-app agents that can execute the underlying code that would otherwise be triggered by a user interacting with the app. These agents then make calls to the same code triggered by user actions, like swipes and taps and pinches. It’s not a bad solution, but the gotcha is that you have to compile these agents into your app while it’s being tested, and you probably want to take it out after the testing is done. The result? You’re not actually testing the code that you release.

The agent approach is a bit odd, too, since the real-world user actions are always from outside the app itself, and so if you’re simulating those actions inside the code rather than triggering them by their corresponding UI action, you’re that much further away from the real experience of an actual human user.

In order to solve this problem and provide for testing with these more real-world style user interactions, both Apple and Google have created user interface automation frameworks for their respective development environments. Google provides uiautomator, which is a Java API for simulating these UI actions on Android devices and emulators, and Apple provides UI Automation Instruments, a JavaScript programming interface for use with iOS devices and simulators.

Appium works by interfacing with these vendor-provided automation frameworks, translating your test code into the platform-specific interactions. We think this is a better approach than using an agent, since you don’t have to compile a test build that contains code which will not be in the production build. You’re best off testing the same code you will release.

The next two points of the Appium philosophy are covered together since they both derive from the decision to implement the WebDriver API and JSON Wire Protocol:

You shouldn’t be locked into a specific language or framework to write and run your tests.

A mobile automation framework shouldn’t reinvent the wheel when it comes to automation APIs.

WebDriver is already well-known as the engine behind Selenium 2 test automation for Web apps. In the Appium implementation, instead of driving a Web browser on a desktop operating system as Selenium does, Appium drives native apps and browsers on mobile operating systems.

Many testers are already very familiar with writing and running tests locally using Selenium WebDriver, where your test will open a browser, run through some interactions with the app, and verify that the test case passes. But the magic of WebDriver really happens when it’s used in a distributed fashion. Instead of making local calls directly to the browser, the WebDriver test becomes an HTTP client and makes requests to a WebDriver server, which in turn actually makes the necessary calls to the browser and app. The elegance of Appium is that it utilizes this framework to interact, not with a desktop browser, but with Android’s UI Automator or Apple’s UIAutomation Instruments, which then perform user actions on the native app or mobile browser running in either an emulator or a real device. And of course Sauce Labs’ infrastructure can manage all of this for you!

(Appium architecture for iOS testing on Sauce)

(Appium architecture for iOS testing on Sauce)

One of the great features of WebDriver is that you can write your test code in your language of choice, since there are WebDriver client libraries for pretty much every popular language out there. The code you write represents the actions users perform on your application under test, and WebDriver can be used with most test runners and frameworks, so chances are your Appium tests can simply be added to your team’s existing workflows.

And so the real-world benefits of the second two points of the Appium philosophy unfold from there. Because Appium implements the WebDriver API, you don’t need to reinvent that wheel. Because WebDriver is so widely used, you can choose any language and test framework and thus incorporate Appium tests easily into your existing process.

A mobile automation framework should be open source, in spirit and practice as well as in name!

Finally, Appium is proudly Open Source and hosted on GitHub. With over 4800 commits, 2000 issues closed, 3400 pull requests, 1100 forks, 1300 stars and dozens of contributors, Appium is a hotbed of excitement and activity in the open source world. The results are spectacular!

And the fun has only just begun!  Look for Appium to continue to evolve rapidly, guided by its core philosophy, and helping you do what you do best, which is delivering great apps to your users!

- Michael Sage, Principal Technology Evangelist, Sauce Labs

Michael Sage is a Principal Technology Evangelist at Sauce Labs who helps software teams develop, deliver, and care for great apps. He’s spent over 15 years as a solutions architect and consultant with software companies including Mercury Interactive, Hewlett Packard, and New Relic. He lives in San Francisco, CA.


Appium + Sauce Labs Integration

August 15th, 2014 by Amber Kaplan

appium_logo‘Appy Friday!

Sauce Labs’ Support Team wants to help our customers easily ramp up their mobile automated testing, so we put together this easy guide to Sauce integration with Appium. We hope you find it to be informative.

If you have any remaining questions about Sauce Labs’ integration with Appium, feel free to leave a note in the comments.

- Cheers, the Support Team at Sauce


Background info:

What is Appium?  Appium is an open-source tool that you can use to automate tests for mobile native, mobile web, and mobile hybrid applications. Just like Selenium Webdriver – which is an open-source tool that you can use to to automate web app tests – Appium is an automation library that you can use to automate tests for mobile applications. Other details:

  • It is cross-platform. This allows you to write tests against multiple platforms (iOS, Android), using the same API.
  • It is not locked into a specific language or framework for writing and running your tests.
  • It was created by extending the Selenium Webdriver JSON Wire Protocol with extra API methods that are useful for mobile automation.
  • Mobile Native Applications vs Mobile Web Applications vs Mobile Hybrid Applications

Types of mobile applications:

  • Mobile Native Applications:  Native apps can use all the features from your device, such as the camera, audio, microphone, and more. These applications work on a specific platform (i.e iOS or Android). Use the respective platform Software Development Kit (SDK) to develop them. Once they’re released for public use, users install the application on their respective devices through an application store (e.g. Google Play or Apple’s App Store).
  • Mobile Web Applications: These ones are a little tricky; instead of being an application that you download on your device, mobile web applications are a mobile website that you can access through a browser on your mobile device (e.g. Safari or Chrome). The advantage to creating and using these is that they are not platform-specific, which means that you should be able to access this application from any mobile platform (i.e. iOS or Android). Unfortunately, these applications have limitations when it comes to using a device’s internal features.
  • Mobile Hybrid Applications: As the name indicates, mobile hybrid applications are part mobile native app and part mobile web app. Just as with mobile native applications, you can find and download mobile hybrid applications using an application store (e.g. Google Play or Apple’s App Store), you can access them through icons on your device, and the app is be able to use all of a device’s features, such as the camera, audio, microphone, etc. In the same manner as a mobile web app, a mobile hybrid application looks like a mobile website that can be accessed through a browser, but in this case the browser is an embedded webview within the application that would just allow to display some HTML. A good example of a mobile hybrid app is that from Bank of America. It provides a user with the mobile native application perks while simply rendering pages from their website.

Appium with Sauce Labs

In certain cases, Sauce Labs uses Appium as the server for our latest iOS simulators and Android emulators. This means that whenever you run a test in Sauce Labs specifying any of the following mobile platforms, Appium is the automation tool in the background that starts and drives the simulator or emulator for your test.

Sauce Labs platforms that use Appium

Mobile Native Applications

  • all iOS platforms
  • all Android platforms

Mobile Web Applications (currently only available for iOS)

  • iPhone 6.1
  • iPhone 7.0
  • iPhone 7.1
  • iPad 6.1
  • iPhone 7.0
  • iPhone 7.1

Start testing with Appium + Sauce Labs


Additional Resources:

Appium Bootcamp: Want to jumpstart automated mobile testing on Sauce with Appium? Check out this bootcamp series by noted Selenium expert Dave Haeffner:

Chapter 1: Get Started
Chapter 2: The Console
Chapter 3: Interrogate Your App
Chapter 4: Your First Test
Chapter 5: Write And Refactor Your Tests
Chapter 6: Run Your Tests
Chapter 7: Automate Your Tests
Chapter 8: Additional Information


Appium Bootcamp – Chapter 8: Additional Information + Resources

August 14th, 2014 by Amber Kaplan

appium_logoThis is the eighth and final post in a series called Appium Bootcamp by noted Selenium expert Dave Haeffner. 

Read:  Chapter 1 Chapter 2 | Chapter 3 | Chapter 4 | Chapter 5 | Chapter 6 | Chapter 7

Dave recently immersed himself in the open source Appium project and collaborated with leading Appium contributor Matthew Edwards to bring us this material. Appium Bootcamp is for those who are brand new to mobile test automation with Appium. No familiarity with Selenium is required, although it may be useful. 

Now that you’re up and running with Appium locally, in the cloud, and on a CI solution, it’s best to show you where you can find more information. Below is a collection of some great resources to help you find your way when it comes to mobile testing.

Community Support

These are the official tutorials for the Appium project for Android and iOS. They served as inspiration and a base for this getting started series. They are great follow-on material since they cover various topics in more depth, and include Java examples as well.

If you have an issue or a question, this is a great place to turn to. Before posting an issue, be sure to read through the Appium Troubleshooting docs and search the group to see if your question has already been asked/answered.

In addition to the Google Discussion Group, you can hop on the Appium HipChat chat room and ask questions from others in the Appium community.

This is a follow-up post answering loads of questions from a webinar from just after thet Appium 1.0 release. It’s chocked full of a lot of great information.

In this video, Jonathan Lipps (Appium’s Chief Architect) explains mobile automation with Appium.

This is an open-source book that is a work in progress; authored by Jonathan Lipps. It’s working title is “Appium: Mobile Automation Made Awesome”.

Some Android Specific Resources

These links (a video, Q&A, and a blog post) cover how Google approaches Android testing.

uiautomator is a crucial component of Android test automation. In this video, the engineers behind it talk about it’s future.

This video is a walk through Google’s newest Android testing framework. This isn’t directly related to Appium, but it contains some useful information.

Some iOS Specific Resources

Appium relies on Apple’s UI Automation support, and these are some solid resources for understanding it better.

Professional Support

If you are a Sauce customer and encounter an issue when using their platform with Appium, be sure to open a support ticket.

If you’re using Appium and you think you’ve found a bug specific to either Android or iOS, then let Google and/or Apple know. In either case it’s best to make sure that the bug is not an Appium issue before filing an issue.

For Google, file an issue here.

For Apple, file an issue here. Apple keeps all bugs private, so it’s worth also filing a duplicate issue here.

Straight To The Source

These are great instructions on how to search through the Appium source code to find more information.

Some Other Resources

There are over 600 Appium questions posted on Stack Overflow for you to peruse.

Xamarin has a free cheat sheet comparing popular mobile app controls. Definitely worth a look.


Now you’re ready, armed with all the information you need to continue your mobile testing journey.

Happy Testing!

Read:  Chapter 1 Chapter 2 | Chapter 3 | Chapter 4 | Chapter 5 | Chapter 6 | Chapter 7

About Dave Haeffner: Dave is a recent Appium convert and the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by thousands of testing professionals) as well as The Selenium Guidebook (a step-by-step guide on how to use Selenium Successfully). He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.

Follow Dave on Twitter - @tourdedave

Re-Blog: Building Markdown-Based Developer Docs

August 13th, 2014 by Amber Kaplan

Sauce Labs developer Chris Wren and his team have been working tirelessly to improve our documentation system, so we thought we’d share what they’ve been up to. Says Chris:

“Recently at Sauce Labs we decided to retool our documentation system. This decision came after accumulating docs in a number of template systems and repos which were difficult to standardize and maintain. The result of this effort was a new markdown-based docs site available at”

For all the details, be sure to check out Chris’ post – just click the image below to view.

Building markdown-based developer docs

Want to contribute to Sauce Labs’ documentation? In the spirit of open source, we’ve housed them in GitHub. Submit away.