Functional Testing | Selenium Testing? Do Cross Browser Testing with Sauce Labs

The official Sauce Labs Blog

Posts Tagged ‘Functional testing’

Run Your Selenium Tests Completely in the Cloud Using Travis CI and Sauce Labs

March 18th, 2013 by Santiago Suarez Ordoñez

Passing builds! This is a quick tutorial to get you started running automated tests completely in the cloud as part of your development process using Travis CI and real browsers in Sauce Labs.

As a guinea pig during this blog post, I used a small, open source Python project of my own called cssify. You can find the cssify build results on Travis and even watch running jobs or videos of the browsers in action in the project’s Open Sauce page.

Steps involved:

Set up your tests to run on Travis CI
Configure Travis CI to start Sauce Connect
Configure your Selenium tests to run on Sauce Labs
Provide additional details to Sauce

What is Travis CI?

Travis CI logo
Travis CI is a hosted continuous integration service for the open source community. It is integrated with GitHub and offers first class support for:
C, C++, Clojure, Erlang, Go, Groovy, Haskell, Java, JavaScript (with Node.js), Perl, PHP, Python, Ruby and Scala.
Their CI environment provides multiple runtimes (e.g. Node.js or PHP versions), data stores and so on. Because of this, hosting your project on Travis means you can effortlessly test web applications against multiple runtimes and data stores without even having all of them installed locally.

Setup Your Tests to Run on Travis CI

There’s multiple tutorials out there to get started using Travis CI. One of my favorite things about Travis is how easy it is to configure and get started.
Check out the official Travis CI docs to get started in your own programming language.

For running your Selenium tests, remember to install the right dependencies, including the Selenium client bindings, like I did in my project. Also, make sure you make Travis start your webapp before the tests start. Your application needs to be serving pages to be able to test it using our browsers :)

Configure Travis CI to Start Sauce Connect

Sauce Connect creates a magical rainbow tunnel that will allow the Sauce browsers driven by your Selenium tests to access the Web Application that your Travis build starts. Once a Sauce Connect tunnel is running, browsers in our cloud can use it to hit localhost, internal IPs or even to just proxy the public internet through the right location.

I’ve created a small Bash script that takes care of downloading, starting and waiting for Connect to be ready. To use Sauce Connect in your build, all you need to do is curl and run this script in your .travis.yml before_script section as I did in my project:

before_script:
  - curl https://gist.github.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash

Configure Your Selenium Tests to Run on Sauce Labs

Once your builds are happily running and Sauce Connect is started automatically by Travis, the next step is having your Selenium scripts run in the Sauce Labs cloud and hit the Application Under Test (AUT) through the tunnel.

To start running real browsers in the Sauce Cloud, make your scripts use Remote WebDriver and point the command executor to localhost:4445 (Sauce Connect will listen in that port and securely and efficiently relay to Sauce). You can find an example of this on the cssify codebase.

self.username = os.environ['SAUCE_USERNAME']
self.key = os.environ['SAUCE_ACCESS_KEY']
hub_url = "%s:%s@localhost:4445" % (self.username, self.key)
self.driver = webdriver.Remote(desired_capabilities=self.caps,
                               command_executor="http://%s/wd/hub" % hub_url)

Now for these browsers to be able to hit the AUT when opening localhost, you need to specify an additional Desired Capability, tunnel-identifier.
The value this capability should be set to is stored in the environment variable TRAVIS_JOB_NUMBER. You can find an example in the cssify codebase for this as well.

self.caps['tunnel-identifier'] = os.environ['TRAVIS_JOB_NUMBER']

Provide Additional Details to Sauce

In addition to running your tests in real browsers using the cloud, there’s additional details you can provide to speed up debugging and navigating test results.
In the cssify tests you can find some examples Desired Capabilities that Sauce uses and that you can pull from Travis CI’s environment variables.

self.caps['build'] = os.environ['TRAVIS_BUILD_NUMBER']
self.caps['tags'] = [os.environ['TRAVIS_PYTHON_VERSION'], 'CI']

That’s it!

You’re set! You are now running your tests using Travis CI, they start a Sauce tunnel on their own and drive real browsers in the cloud. Time to start thinking about running these in parallel to speed up your builds now!

Here are examples of what your Travis CI build should look and what you’ll see from the Sauce side.

Posted in Software Development & Testing

1 Comment »

Upcoming Selenium Events: Page Object Model, Optimizing Test Performance, and More

June 21st, 2012 by Ashley Wilson

June and July are busy months for the Selenium and Sauce community. Check out these meetups happening across the country (and world), and let us know if we’re missing anything. Hope to see you there!

June 21: PDX Selenium Meetup
Beyond Page Object Patterns
By Sam Woods, Software Development Engineer in Test at WebTrends
Learn about the evolution of automated UI tests from “scripts” to fully functional, maintainable, reliable abstracted automated test cases using the page object pattern.

June 27: Colombo Selenium Meetup
Selenium’s Journey
By Jason Huggins, Co-Founder of Sauce Labs
Get an introduction to Selenium’s colorful history in this virtual meetup with Jason Huggins, original creator of Selenium.

July 9th: Boston Selenium Meetup
July 11th: DC Selenium Meetup
Optimizing Selenium for Executed Performance
By Santiago Suarez Ordoñez, Selenium Ninja at Sauce Labs
This talk will cover test writing tips gathered over the years from helping hundreds of Selenium users and engineers make their tests run faster and more reliably, independently of the technology stack used. We’ll be talking about things such data independence, making atomic tests and generating application state.

July 10th: Continuous Delivery NYC Meetup
Continuous Delivery at New Relic
By: Bjorn Freeman-Benson, VP of Engineering at New Relic
Bjorn will talk about New Relic’s experiences in growing from a few developers supporting a few thousand users to 70+ developers delivering continuously to thousands of large and small commercial customers.

July 17th: San Jose Selenium Meetup
HARdy Har Har: How to generate HTTP Archive Files
By Adam Goucher, Selenium Expert and Consultant at Element 34
This talk will show you how to generate HTTP Archive (HAR) files in two different ways; using a custom Firefox WebDriver profile and with the BrowserMob Proxy.

July 18th: San Diego Web Performance Meetup
Capturing Performance Data from your Selenium Tests
By Patrick Lightbody, Director of Product Management at Neustar Enterprise Services
Learn from one of the original Selenium developers how you can use open source projects and new specifics, such as Navigation Timings, to augment your existing test suite and capture performance in your existing continuous integration environment.

July 18th: OC PHP Meetup
Selenium: Less Testing and More Coding
By Jonathan Lipps, Front-End Engineer at Sauce Labs
Jonathan will give a demo of a basic PHP web application and go through the motivations for using Selenium as a way to consistently and reliably test the application. He’ll then describe how Sauce Labs’s cloud service enables testing to transparently scale to multiple browsers and platforms and reduce several frustrations with running tests locally.

Posted in Events

No Comments »

Javascript + Selenium: The Rockstar Combination of Testing

July 25th, 2011 by Ashley Wilson

For our July Selenium meetup, held last Thursday, we wanted to give attendees something a little different to chew on. Thanks to our good friends at Yammer, who co-hosted the event with us, we did so not only with delicious catered Mexican food, but also plenty of Javascript & Selenium testing goodness to go around.

Bob Remeika, senior engineer at Yammer, gave a spirited presentation that left no one questioning his stance on testing (his opening slide – “Test your shit” – really said it all). He gave us an inside look at how Yammer tests using a combination of Jellyfish and Sauce OnDemand, and gave some great advice on knowing what and how to test when you’re just starting out.

We also had Adam Christian, Sauce Labs’ Javascript Aficionado and the creator of Jellyfish, give two talks. The first, a lightning talk titled “Javascript Via Selenium: The Good, The Bad, The Obvious”, covered some of the lesser known things about Javascript testing via Selenium.

The second showed off how you can use Jellyfish, the open source Javascript runner that he announced a few weeks ago, to run your JS unit tests in any environment.

Thanks to Adam, Bob, and Yammer for making this quite the fun and memorable meetup. As always, the San Francisco Selenium Meetup group is free to join & we meet monthly at different venues around the Bay Area to talk all things testing. See you in August!

Posted in Events, Selenium, Videos

No Comments »

Sauce Builder Webinar: From zero to creating, storing and running automated tests in under 30 mins

May 3rd, 2011 by The Sauce Labs Team

As you may have heard, we recently released a new, free tool that makes building and running Selenium tests easier than before. Sauce Builder is now available for download, and with it comes the ability to create Selenium tests in your Firefox browser by simply clicking around your application. You can then run those tests in Sauce OnDemand with just a few clicks, no Selenium expertise necessary.

To explain more about Sauce Builder and how automated testing can accelerate your development velocity, we held a webinar with Adam Christian, a Sauce Labs developer and project lead for Sauce Builder.

With Sauce Builder, we’re hoping to simplify and improve the cross-browser testing process for development and QA teams. Specifically, Sauce Builder will allow you to:

Build Selenium tests with zero programming - Simply click through your application and Sauce Builder writes a Selenium scripts that reflect your actions.
Export results in the language of your choice - HTML, JAVA, Groovy, C#, Perl, PHP, Python and Ruby so your tests speak the same language as your application and dev team.
Eliminate bugs faster - Use immediate video playback of your tests in action, and share them with your teammates.
Remove test infrastructure headaches - Sauce Builder makes it a snap to either run tests locally in Firefox or in the cloud with access to all the browser / OS combinations supported in the super scalable Sauce OnDemand service.

Register To See The Webinar Now!

Posted in News & Product Updates, Videos

No Comments »

Sauce Labs Eliminates Barrier to Automated Cross-Browser Testing with Sauce Builder

March 31st, 2011 by The Sauce Labs Team

New tool enables QA pros to build and run Selenium tests without Selenium expertise or infrastructure

SAN FRANCISCO, CA — (Mar. 31, 2011) – Sauce Labs, the web application testing company, today introduced Sauce Builder, a free testing tool that makes it easy for users to build Selenium tests without Selenium expertise and run them with the Sauce OnDemand service. Sauce Builder allows users to build automated Selenium tests simply by clicking through an application. By eliminating the complexity of hand-coding Selenium scripts, Sauce Builder accelerates the adoption of automated testing for QA and development teams that have been craving the development productivity Selenium offers. Sauce Builder is free and available for immediate download.

“Automated testing has historically been one of the most complicated, yet most valuable, technologies for companies that build software. Automated testing is doubly challenging because teams need to build and maintain a testing environment and on top of that, building tests can require significant technical skill,” said John Dunham, CEO of Sauce Labs. “We launched our Sauce OnDemand cloud service last year to eliminate the headache of maintaining a test infrastructure. Now with Builder, we’ve removed the next barrier to the adoption of automated testing and we’re very excited to see how this combination can help QA and development teams achieve their goals.”

With Sauce Builder, Sauce Labs continues to simplify and improve the cross-browser testing process for development and QA teams. Sauce Builder’s benefits include:

Build Selenium tests with zero programming – Simply click through your application and Sauce Builder writes a Selenium scripts that reflect your actions
Export results in the language of your choice – HTML, Java, Groovy, C#, Perl, PHP, Python and Ruby so your tests speak the same language as your application and dev team
Eliminate bugs faster – Use immediate video playback of your tests in action, and share them with your teammates
Remove test infrastructure headaches - Sauce Builder makes it a snap to either run tests locally in Firefox or in the cloud with access to all the browser / operating system combinations supported in the super scalable Sauce OnDemand service

With over four million downloads in just four years, the Selenium project is the world’s most popular functional testing framework for web applications. Designed to further expand Selenium adoption, Sauce Builder is the first web-based Selenium tool of its kind, including technology Sauce Labs acquired from Go Test It in 2010. After becoming more familiar with the technology post-acquisition, Sauce Labs elected to open source the code under the name “Se Builder” earlier this year because the technology held so much promise for the Selenium community.

Sauce Builder expands the capabilities of Se Builder by enabling users to directly access Sauce OnDemand, the cloud-based Selenium service, to run their tests. Sauce OnDemand is free to try for up to 200 testing minutes every month. Sauce Labs is also leading a collaborative effort with the Selenium community to deliver a new generalized plug-in architecture for Se Builder that among other things will support integrated plug-ins for testing services like Sauce OnDemand.

To learn more about how automated testing can accelerate your development velocity, please join Adam Christian,a Sauce Labs developer and project lead for Sauce Builder, on Tuesday, April 19^th at 10AM Pacific for the webinar, “From zero to creating, storing and running automated tests in under 30 minutes”.

“Debugging takes up valuable time that developers could be using to focus on their applications,” said Sauce Labs’ Christian. “Now with Sauce Builder, developers can leverage this great development environment through our cloud testing infrastructure and not worry about dealing with building or maintaining their own costly testing infrastructure.”
About Sauce Labs
Sauce Labs, web application testing company, provides Sauce OnDemand, a cloud based service that allows users to run automated cross-browser functional tests faster and eliminating the need to maintain their own test infrastructure. To date, over four million Sauce OnDemand tests have been run in the Sauce cloud. The lead investor of Sauce Labs is the Contrarian Group, Peter Ueberroth’s investment management firm. Sauce Labs is headquartered in San Francisco, California. For more information, visit http://saucelabs.com.

Media Contact
Chantal Yang
LEWIS Pulse for Sauce Labs
sauce@lewispulse.com
415-875-7494

Posted in News & Product Updates, Sauce Product Info

No Comments »

Selenium 2 Webinar: The Next Generation of Web and Mobile Application Testing

February 10th, 2011 by Ashley Wilson

Curious to know more about Selenium 2? Join Jason Huggins, creator of Selenium, for a webinar devoted to covering the essentials of this new testing tool that combines the best of WebDriver and Selenium.

Here’s a run-down of what you will learn next Wednesday, 2/16, at 10am PST:

iPhone and Android testing – See how Selenium 2 allows you to use built-in Android and iPhone emulators to test versions of your applications on the most popular mobile platforms.
Cleaner API for IDE users – Selenium introduces a simplified interface for IDE users that directs you to focus on only two basic objects to construct tests: WebDriver (browsers) and WebElements (anything on a web page). In Selenium 2, every API library is now tailored to each programming language for easier usability.
Enhanced scalability – The new Selenium 2 architecture allows developers and QA teams to “scale up and down”. For a single test on a local machine, you no longer need a background server. But when you want to scale up to run tests across multiple machines with multiple browser configurations, Selenium has all the power you’ll need.
Improved architecture – The new Selenium 2 architecture has enabled the introduction of a number of features that developers and QA pros will love, including native keyboard, support for mouse events, improved capabilities for handling pop-ups, and more stable browser control.

To join us, please click here to register. If you were wondering if Sauce Labs supports Selenium 2 testing the cloud, the answer is yes :-)

See you next week!

Posted in Events, Selenium

5 Comments »

Advanced Selenium Synchronization with ‘Latches’

February 7th, 2011 by Adam Goucher

Synchronization with Selenium is the number two problem I see with people’s scripts (the first being broken locators). This is actually a pretty easy problem to solve in all but the most pathological situations.

Web 1.0
Remember the good ol’ days when clicking a link caused the browser to fetch a new page? Those were, and remain, super easy to synchronize with using waitForPageToLoad and its language specific variants. Here’s an example of this in Ruby:

@selenium.click "link=click here", :wait_for => :page

Web 2.0
But along came Web 2.0 and all its AJAX-y ‘goodness’. No longer could we be guaranteed that some action or event on the page would trigger a page load. For that, we needed to use a combination of isElementPresent or isVisible.

@selenium.click "link=click here", :wait_for => :element, :element => "my locator"
@selenium.click "link=click here", :wait_for => :visible, :element => "my locator"

While I haven’t done it yet for Ruby, I often combine the two together to create an ‘available’ condition to make sure that the element I care about is both present and visible. Something like this.

@selenium.click "link=click here", :wait_for => :available, :element => "my locator"

Post Web 2.0
Some AJAX calls affect multiple elements and are both tricky and/or time consuming to figure out all the places to watch. New technologies like COMET also make element availability more fallible than it might originally be. For these situations you need to use a ‘latch’ for synchronization.

A what?

The term ‘latch’ in this case is a value that is set in the browser’s DOM that your script monitors for synchronization rather than something in the actual page. And example is in order. Here is an AJAX call in Ruby on Rails.

<%= link_to_remote( "click here",
                   :update => "time_div",
                   :url => { :action => :say_when },
                   :before => "window.latch = 'started'",
                   :complete => "window.latch = 'done'") %>

What this will do is set window.latch to the string value ‘started’ in the DOM before it actually executes and will set it to ‘done’ after it is complete. No problem. People who work with AJAX are used to working with callbacks. The twist happened in our Selenium script.

@selenium.click "link=click here", :wait_for => :condition, :javascript => "window.latch == 'done'"

Now instead of checking for an element to be present or visible, we are waiting via waitForCondition for the latch conditions; in this case, ‘window.latch is done’.

I consider the latch technique as the synchronization method of last resort since it requires changing production code to support automation. But successful automation often calls for just that.

Posted in Selenium

3 Comments »

Trends in Testing: Continuous Integration, Mobility, Open Source, Cloud

May 10th, 2010 by The Sauce Labs Team

Functional testing is the automation of web app testing across several platforms or browsers. In the past, functional testing was less common due to long development cycles and the lack of multiple browser options. It was done primarily in-house via proprietary software like Rational or QuickTestPro.

Today, there are four trends converging to radically change how organizations test web applications:

Continuous integration – An increased emphasis on performance of the software through the end customer’s perspective is leading agile development. Gartner predicts that by 2012 agile development methodologies will be used by 80 percent of all software development projects. Teams are shifting away from rigid quality control to quality that is demonstrable to the end user. With this shift, comes a need for quick, simple and automated testing tools.
Open source – Open source tools, like Selenium, are viable for functional testing. Today’s development environment has dramatically changed and a lot more people are contributing than ever before. The major advantages of open-source are speed, time to market, time to value, and the ability to reach and grow a developer and user community. In fact, Gartner predicts that by 2012, 80 percent of all commercial software will include elements of open source technology.
Cloud computing – Testing in the cloud is an affordable and scalable alternative to testing behind a firewall. It is expensive and time consuming to maintain on site test infrastructures that cover a vast number of browsers (and versions) of operating systems in several languages. The increase in the adoption of cloud computing creates an opportunity to leverage the space for functional testing. We believe the life cycle for cloud-based applications will look different in the future. New solutions that are cloud based will support cloud-base applications.
Mobility – Websites need to support multiple browsers such as Internet Explorer, Firefox, Chrome, Rockmelt, Opera on various platforms including Windows and Mac. Smartphones only add to this “Browser War 2.0” battle with the plethora of mobile operating systems. These various environments make the infrastructure required more complex and the need for functional testing more relevant that ever before.

Are there other trends beyond continuous integration, mobility, open source and the cloud that will change how organizations test their web applications?

Posted in Software Development & Testing

5 Comments »

Highlights from our April 20th Selenium Testing Tools Demo Night

April 29th, 2010 by Ashley Wilson

Below are clips from our most recent Selenium meetup, which drew quite an enthusiastic crowd. Presenters Ray Vizzone of Test Labs, Inc and Matt Krapivner of Smart Pilot, David Vydra of Guideware, and Quality Tree founder Elisabeth Hendrickson were on hand at the Sauce Labs office to demonstrate leading edge tools / techniques that make functional testing easier and more effective. Also featured was a special Flex Pilot demo by Sauce Labs’ own Adam Christian.

Make sure to visit our #SFSE Meetup page here to RSVP for our next event on May 19, 2010 at Mozilla’s headquarters in Mountain View. Speakers from Mozilla and LinkedIn will discuss how they use Selenium and other open-source tools to handle quick and effective web testing. Take it from us – you don’t want to miss this!

Introduction to 4/20 Selenium Meetup with Stephen Donner of Mozilla