Posts Tagged ‘Sauce Labs’

How To Add Visual Testing To Existing Selenium Tests

February 27th, 2015 by Dave Haeffner

Thanks again to those of you who attended our recent webinar with Applitools on automated visual testing.  If you want to share it or if you happened to miss it, you can catch the audio and slides hereWe also worked with Selenium expert Dave Haeffner to provide the how-to on the subject. Enjoy his post below.

 

The Problem

In previous write-ups I covered what automated visual testing is and how to do it. Unfortunately, based on the examples demonstrated, it may be unclear how automated visual testing fits into your existing automated testing practice.

Do you need to write and maintain a separate set of tests? What about your existing Selenium tests? What do you do if there isn’t a sufficient library for the programming language you’re currently using?

A Solution

You can rest easy knowing that you can build automated visual testing checks into your existing Selenium tests. By leveraging a third-party platform like Applitools Eyes, this is a simple feat.

And when coupled with Sauce Labs, you can quickly add coverage for those hard to reach browser, device, and platform combinations.

Let’s step through an example.

An Example

NOTE: This example is written in Java with the JUnit testing framework.

Let’s start with an existing Selenium test. A simple one that logs into a website.

// filename: Login.java

import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class Login {

    private WebDriver driver;

    @Before
    public void setup() {
        driver =  new FirefoxDriver();
    }

    @Test
    public void succeeded() {
        driver.get("http://the-internet.herokuapp.com/login");
        driver.findElement(By.id("username")).sendKeys("tomsmith");
        driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");
        driver.findElement(By.id("login")).submit();
        Assert.assertTrue("success message should be present after logging in",
                driver.findElement(By.cssSelector(".flash.success")).isDisplayed());
    }

    @After
    public void teardown() {
        driver.quit();
    }
}

In it we’re loading an instance of Firefox, visiting the login page on the-internet, inputting the username & password, submitting the form, asserting that we reached a logged in state, and closing the browser.

Now let’s add in Applitools Eyes support.

If you haven’t already done so, you’ll need to create a free Applitools Eyes account (no credit-card required). You’ll then need to install the Applitools Eyes Java SDK and import it into the test.

// filename: pom.xml

<dependency>
  <groupId>com.applitools</groupId>
  <artifactId>eyes-selenium-java</artifactId>
  <version>RELEASE</version>
</dependency>
// filename: Login.java

import com.applitools.eyes.Eyes;
...

Next, we’ll need to add a variable (to store the instance of Applitools Eyes) and modify our test setup.

// filename: Login.java
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;

    @Before
    public void setup() {
        WebDriver browser =  new FirefoxDriver();
        eyes = new Eyes();
        eyes.setApiKey("YOUR_APPLITOOLS_API_KEY");
        driver = eyes.open(browser, "the-internet", "Login succeeded");
    }
...

Rather than storing the Selenium instance in the driver variable, we’re now storing it in a localbrowser variable and passing it into eyes.open — storing the WebDriver object that eyes.openreturns in the driver variable instead.

This way the Eyes platform will be able to capture what our test is doing when we ask it to capture a screenshot. The Selenium actions in our test will not need to be modified.

Before calling eyes.open we provide the API key (which can be found on your Account Details page in Applitools). When calling eyes.open, we pass it the Selenium instance, the name of the app we’re testing (e.g., "the-internet"), and the name of the test (e.g., "Login succeeded").

Now we’re ready to add some visual checks to our test.

// filename: Login.java
...
    @Test
    public void succeeded() {
        driver.get("http://the-internet.herokuapp.com/login");
        eyes.checkWindow("Login");
        driver.findElement(By.id("username")).sendKeys("tomsmith");
        driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");
        driver.findElement(By.id("login")).submit();
        eyes.checkWindow("Logged In");
        Assert.assertTrue("success message should be present after logging in",
                driver.findElement(By.cssSelector(".flash.success")).isDisplayed());
        eyes.close();
    }
...

With eyes.checkWindow(); we are specifying when in the test’s workflow we’d like Applitools Eyes to capture a screenshot (along with some description text). For this test we want to check the page before logging in, and then the screen just after logging in — so we use eyes.checkWindow(); two times.

NOTE: These visual checks are effectively doing the same work as the pre-existing assertion (e.g., where we’re asking Selenium if a success notification is displayed and asserting on the Boolean result) — in addition to reviewing other visual aspects of the page. So once we verify that our test is working correctly we can remove this assertion and still be covered.

We end the test with eyes.close. You may feel the urge to place this in teardown, but in addition to closing the session with Eyes, it acts like an assertion. If Eyes finds a failure in the app (or if a baseline image approval is required), then eyes.close will throw an exception; failing the test. So it’s best suited to live in the test.

NOTE: An exceptions from eyes.close will include a URL to the Applitools Eyes job in your test output. The job will include screenshots from each test step and enable you to play back the keystrokes and mouse movements from your Selenium tests.

When an exception gets thrown by eyes.close, the Eyes session will close. But if an exception occurs before eyes.close can fire, the session will remain open. To handle that, we’ll need to add an additional command to our teardown.

// filename: Login.java
...
    @After
    public void teardown() {
        eyes.abortIfNotClosed();
        driver.quit();
    }
}

eyes.abortIfNotClosed(); will make sure the Eyes session terminates properly regardless of what happens in the test.

Now when we run the test, it will execute locally while also performing visual checks in Applitools Eyes.

What About Other Browsers?

If we want to run our test with it’s newly added visual checks against other browsers and operating systems, it’s simple enough to add in Sauce Labs support.

NOTE: If you don’t already have a Sauce Labs account, sign up for a free trial account here.

First we’ll need to import the relevant classes.

// filename: Login.java
...
import org.openqa.selenium.Platform;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
...

We’ll then need to modify the test setup to load a Sauce browser instance (via Selenium Remote) instead of a local Firefox one.

// filename: Login.java
...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", "Login succeeded");
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                "YOUR_SAUCE_USERNAME",
                "YOUR_SAUCE_ACCESS_KEY");
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        eyes = new Eyes();
        eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));
        driver = eyes.open(browser, "the-internet", "Login succeeded");
    }
...

We tell Sauce what we want in our test instance through DesiredCapabilities. The main things we want to specify are the browser, browser version, operating system (OS), and name of the test. You can see a full list of the available browser and OS combinations here.

In order to connect to Sauce, we need to provide an account username and access key. The access key can be found on your account page. These values get concatenated into a URL that points to Sauce’s on-demand Grid.

Once we have the DesiredCapabilities and concatenated URL, we create a Selenium Remote instance with them and store it in a local browser variable. Just like in our previous example, we feedbrowser to eyes.open and store the return object in the driver variable.

Now when we run this test, it will execute against Internet Explorer 8 on Windows XP. You can see the test while it’s running in your Sauce Labs account dashboard. And you can see the images captured on your Applitools account dashboard.

A Small Bit of Cleanup

Both Applitools and Sauce Labs require you to specify a test name. Up until now, we’ve been hard-coding a value. Let’s change it so it gets set automatically.

We can do this by leveraging a JUnit TestWatcher and a public variable.

// filename: Login.java
...
import org.junit.rules.TestRule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;
    public String testName;

    @Rule
    public TestRule watcher = new TestWatcher() {
        protected void starting(Description description) {
            testName = description.getDisplayName();
        }
    };
...

Each time a test starts, the TestWatcher starting function will grab the display name of the test and store it in the testName variable.

Let’s clean up our setup to use this variable instead of a hard-coded value.

// filename: Login.java
...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", testName);
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                System.getenv("SAUCE_USERNAME"),
                System.getenv("SAUCE_ACCESS_KEY"));
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        eyes = new Eyes();
        eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));
        driver = eyes.open(browser, "the-internet", testName);
    }
...

Now when we run our test, the name will automatically appear. This will come in handy with additional tests.

One More Thing

When a job fails in Applitools Eyes, it automatically returns a URL for it in the test output. It would be nice if we could also get the Sauce Labs job URL in the output. So let’s add it.

First, we’ll need a public variable to store the session ID of the Selenium job.

// filename: Login.java
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;
    public String testName;
    public String sessionId;
...

Next we’ll add an additional function to TestWatcher that will trigger when there’s a failure. In it, we’ll display the Sauce job URL in standard output.

// filename: Login.java
...
    @Rule
    public TestRule watcher = new TestWatcher() {
        protected void starting(Description description) {
            testName = description.getDisplayName();
        }

        @Override
        protected void failed(Throwable e, Description description) {
            System.out.println(String.format("https://saucelabs.com/tests/%s", sessionId));
        }
    };
...

Lastly, we’ll grab the session ID from the Sauce browser instance just after it’s created.

// filename: Login.java
...
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        sessionId = ((RemoteWebDriver) browser).getSessionId().toString();
...

Now when we run our test, if there’s a Selenium failure, a URL to the Sauce job will be returned in the test output.

Expected Outcome

  • Connect to Applitools Eyes
  • Load an instance of Selenium in Sauce Labs
  • Run the test, performing visual checks at specified points
  • Close the Applitools session
  • Close the Sauce Labs session
  • Return a URL to a failed job in either Applitools Eyes or Sauce Labs

Outro

Happy Testing!

 

About Dave Haeffner: Dave is the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by hundreds of testing professionals) as well as a new book, The Selenium Guidebook. He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.

Sauce Labs Appoints Technology Veteran Charles Ramsey as Company’s First Chief Revenue Officer

February 24th, 2015 by Amber Kaplan

SAN FRANCISCO, CA-  Sauce Labs, Inc., the leading cloud-based web and mobile application testing platform, today announced that it has appointed Charles Ramsey as the company’s first Chief Revenue Officer (CRO). Ramsey brings more than 25 years of industry experience and insight to his role at Sauce Labs. He will be responsible for all customer-facing areas, including sales, business development, and marketing, as well as continuing to build on Sauce Labs record 2014 results as the company extends its leadership position in the booming automated testing market.

“Charles is an innovative strategist, committed to building strong relationships with customers and partners,” said Jim Cerna, CEO of Sauce Labs. “His demonstrated ability to strategically grow companies will help us address the exploding demand we are seeing in the market for our technology. With Charles’ experience, leadership, and track record of repeated successes, we are poised to continue our rapid growth trajectory through 2015 and beyond.”

“Sauce Labs is well-positioned to take advantage of the dramatic proliferation of web and mobile apps across a variety of devices and operating systems. I look forward to working with the entire Sauce team to further the company’s market leadership and growth by bringing new levels of innovation, customer experience, and value to the marketplace,” said Ramsey.

Sauce Labs provides an instantly scalable testing cloud that is optimized for continuous integration (CI) and continuous delivery (CD). When tests are automated and run in parallel on virtual machines across multiple browsers and platforms, testing time is reduced and developer time is freed up from managing infrastructure. When paired with a CI system, developers can easily test web, hybrid and native applications early on in their development cycles, continuously and affordably. Sauce Labs currently supports more than 480 browser, operating system and device platform combinations.

Prior to joining Sauce Labs, Ramsey was an early member of the Quest Software management team, where he served as vice president of World Wide Marketing and Sales. He is a former Venture Partner at JMI Equity and has also served on the Board of Directors at notable companies such as Configuresoft, Inc. and ServiceNow, Inc. Early in his career, Ramsey rose to vice president, North America Sales for Computer Intelligence, a division of Ziff Davis, after beginning his career with the IBM National Accounts Division in a variety of sales assignments. He has a Bachelor of Arts from the University of California, San Diego and a Master of Information Management from the American Graduate School of International Management.

Helpful Links

About Sauce Labs
Sauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity and reduce infrastructure costs for software teams of all sizes.

Sauce Labs is a privately-held company funded by Toba Capital, Salesforce Ventures, Triage Ventures and the Contrarian Group. For more information, please visit http://saucelabs.com.

Automated Testing News: Link Round-Up

January 30th, 2015 by Amber Kaplan
Happy Friday! Here’s a quick round-up of some pieces around the web. This week, learn about some on tools for checkout optimization, how to leverage automated UI testing on Sauce with Grunt and Intern, and get more QTP/UFT VS Selenium resources.  See below for snippets and links to the full articles.

(more…)

Cross-Browser Layout Testing With Galen Framework And Sauce Labs

January 8th, 2015 by Amber Kaplan

Every web developer faces this problem once the website is implemented: cross-browser layout testing. This is a very annoying task as you have to look at your website on a lot of different browsers. We used to perform a manual check on all these browsers at the end of website development, and as a result we get some layout issues and have to find hacks to fix these deffects. But what if we could automate layout testing of the website and perform it always in all major browsers? Imagine you follow a TDD technique in your front-end development and you want to have an early feedback on layout issues. To solve this problem you could have a look at Galen Framework.

In short Galen Framework is a layout and functional testing framework that has its own language for describing the layout. The way you test layout using this approach is by declaring a set of elements on the page and rules describing where each element is expected to be. The elements are being tested relatively to each other. This way you can make your tests very flexible and run them on different browsers and even mobile devices in case your website is responsive.

In this article I would like to show you how to write a basic layout test and run it in the SauceLabs cloud. SauceLabs offers various platforms and browsers and even allows you to choose browser versions. Lets take the major browsers like: Internet Explorer 11, Firefox, Chrome, Opera and Safari on Mac. The website under test will be http://testapp.galenframework.com.

(more…)

7 Tips For Maximizing Your Sauce Experience

January 6th, 2015 by lauren nguyen

Here at Sauce, we’re dedicated to helping teams bring quality applications to market faster and more cost-effectively. Our customers use Sauce in a variety of ways based on their development and deployment processes and setups, but there are a few practices that are common to a significant number of our users who run automated tests. We’ll talk about what 7 of these best practices are and their benefits. (more…)

We Automated 12 Android Phones To Sing ‘Appy Holidays To You [VIDEO]

December 23rd, 2014 by Amber Kaplan

With Holiday Season and the close of the fiscal year approaching, we brought in all of our remote workers to spend some time together. We had the obligatory corporate holiday party, a late-night LAN party, and countless, invaluable meetings that laid the foundation for software development projects in the new year.

Our VP of Engineering also arranged our first hackathon, in which we submitted ideas for programming projects that were tangentially related to our business. To participate, we were broken into teams and powered through our ideas and execution over two days.

Some of our developers have been interacting with Android phones, but haven’t had a chance to play with them. Thus, I ended up on a team of three with the goal of getting multiple Android devices to sing in harmony.

We’re happy to share the result of our efforts with this cheerful holiday video.

See the original video here. Follow the conversation on Reddit here and here and on HackerNews here.

So, how does it work?

We went with a pretty quick-and-dirty approach, seeing as this was a two-day event. Having multiple devices sync musical notes in realtime was definitely out of the question, so we opted for seeing if we could coordinate the phones to begin playing their individuals parts of a song at the same time.

Android devices are finicky and difficult to coordinate, many of their operations take varying amounts of time, even on identical devices. Simply pushing a song to each of them and telling them to play, results in some phones playing within a second while others can be up to four seconds behind.

Here’s what we did:

@classam wrote a python script which takes any MIDI file and distributes its separate instrumental parts onto an arbitrary number of tracks. It’s pretty neat: specify just two tracks, and half the parts of the song will be played on one track, and half on the other. If you specify more tracks than available parts, it copies the more significant parts onto the leftover tracks so every phone will feel like it’s a valued member of the choir.

I wrote an Android app which runs on the phones. The app runs on each phone and listens on a socket. When it gets the ‘GO’ command over the USB cable, it plays the music file it’s been given. It also displays a musical visualization of the sound it’s playing so you can match up the phones to the sounds you hear.

@etmillin wrote a javascript program which finds all the Android devices connected to a computer, gets a connection to the song App running on each one, and sends the ‘GO’ command to all of them at once.

When you glue our three pieces together, the following happens:

  • A MIDI song gets broken into parts, one for each phone
  • The parts of the song get uploaded, one to each phone
  • The song-playing app gets started on all the phones
  • The ‘GO’ command gets sent to all the phones at once

Now they all play together!

Other hackathon projects included experimenting with different types of network messaging solutions, devOps management tools, ECMAscript7, contributing to open source projects like Travis CI, and building hardware which displays the state of our continuous integration build.

Be warned, we didn’t polish the code for this blog post but you can find it on github: https://github.com/classam/5rat

-Jonah Stiennon, Ecosystems, Sauce Labs

Want to work at Sauce Labs? Submit your resume! https://saucelabs.com/careers

Mobile Testing Talk: What’s New With Appium [RECAP]

December 19th, 2014 by Amber Kaplan

appium_logo_with_sauce_NEW

Thanks for joining us last week for our latest webinar, Mobile Testing Talk: What’s New With Appium, which featured Jonathan Lipps, the Chief Architect for the Appium Project.

To recap, Jonathan gave listeners a tour of Appium version 1.3.x, including the stability improvements and features the team has added since the Appium 1.0 release back in May of 2014. He also touched on the following:

  •  Appium 1.3.x release features and improvements
  •  New platforms
  •  Better hybrid support
  •  Examples of maturing Appium clients & more

We had great feedback on the content and an influx of interesting questions that got addressed in the Q&A at the end.

Missed the webinar, need to hear it again, or want share it with your team? Check out the slides below (more…)

Mobile Testing Talk: What’s New With Appium [WEBINAR]

December 10th, 2014 by Amber Kaplan

appium_logo_with_sauce_NEW

Developed by Sauce Labs and a thriving community of open source contributors, Appium is a cross-platform automation framework for testing mobile web, native, and hybrid applications. (more…)

Sauce Connect Gets a Speed Boost & WebSocket Support

November 19th, 2014 by Justin Raczak

Sauce Connect was designed with security as priority one. But given this technology’s critical position in your testing process we know that performance and utility are important, too. For that reason we have made two major improvements to Sauce Connect tunnels. (more…)

Coming Soon – A Reimagined Sauce Labs UI

November 19th, 2014 by Justin Raczak

We are passionate about building products and services that help our users maximize the value they get out of their continuous integration and continuous delivery workflows. And while our core products serve this mission well, especially if you have integrated your CI server and are passing us test statuses, we realized we can do even more. We are excited to announce that we have begun work to completely overhaul the Sauce Labs UI and create a new experience specially designed for CI/CD workflows. The new UI will begin rolling out in phases next month. (more…)