How To Add Visual Testing To Existing Selenium Tests

February 27th, 2015 by Dave Haeffner

Thanks again to those of you who attended our recent webinar with Applitools on automated visual testing.  If you want to share it or if you happened to miss it, you can catch the audio and slides hereWe also worked with Selenium expert Dave Haeffner to provide the how-to on the subject. Enjoy his post below.

 

The Problem

In previous write-ups I covered what automated visual testing is and how to do it. Unfortunately, based on the examples demonstrated, it may be unclear how automated visual testing fits into your existing automated testing practice.

Do you need to write and maintain a separate set of tests? What about your existing Selenium tests? What do you do if there isn’t a sufficient library for the programming language you’re currently using?

A Solution

You can rest easy knowing that you can build automated visual testing checks into your existing Selenium tests. By leveraging a third-party platform like Applitools Eyes, this is a simple feat.

And when coupled with Sauce Labs, you can quickly add coverage for those hard to reach browser, device, and platform combinations.

Let’s step through an example.

An Example

NOTE: This example is written in Java with the JUnit testing framework.

Let’s start with an existing Selenium test. A simple one that logs into a website.

// filename: Login.java

import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class Login {

    private WebDriver driver;

    @Before
    public void setup() {
        driver =  new FirefoxDriver();
    }

    @Test
    public void succeeded() {
        driver.get("http://the-internet.herokuapp.com/login");
        driver.findElement(By.id("username")).sendKeys("tomsmith");
        driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");
        driver.findElement(By.id("login")).submit();
        Assert.assertTrue("success message should be present after logging in",
                driver.findElement(By.cssSelector(".flash.success")).isDisplayed());
    }

    @After
    public void teardown() {
        driver.quit();
    }
}

In it we’re loading an instance of Firefox, visiting the login page on the-internet, inputting the username & password, submitting the form, asserting that we reached a logged in state, and closing the browser.

Now let’s add in Applitools Eyes support.

If you haven’t already done so, you’ll need to create a free Applitools Eyes account (no credit-card required). You’ll then need to install the Applitools Eyes Java SDK and import it into the test.

// filename: pom.xml

<dependency>
  <groupId>com.applitools</groupId>
  <artifactId>eyes-selenium-java</artifactId>
  <version>RELEASE</version>
</dependency>
// filename: Login.java

import com.applitools.eyes.Eyes;
...

Next, we’ll need to add a variable (to store the instance of Applitools Eyes) and modify our test setup.

// filename: Login.java
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;

    @Before
    public void setup() {
        WebDriver browser =  new FirefoxDriver();
        eyes = new Eyes();
        eyes.setApiKey("YOUR_APPLITOOLS_API_KEY");
        driver = eyes.open(browser, "the-internet", "Login succeeded");
    }
...

Rather than storing the Selenium instance in the driver variable, we’re now storing it in a localbrowser variable and passing it into eyes.open — storing the WebDriver object that eyes.openreturns in the driver variable instead.

This way the Eyes platform will be able to capture what our test is doing when we ask it to capture a screenshot. The Selenium actions in our test will not need to be modified.

Before calling eyes.open we provide the API key (which can be found on your Account Details page in Applitools). When calling eyes.open, we pass it the Selenium instance, the name of the app we’re testing (e.g., "the-internet"), and the name of the test (e.g., "Login succeeded").

Now we’re ready to add some visual checks to our test.

// filename: Login.java
...
    @Test
    public void succeeded() {
        driver.get("http://the-internet.herokuapp.com/login");
        eyes.checkWindow("Login");
        driver.findElement(By.id("username")).sendKeys("tomsmith");
        driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");
        driver.findElement(By.id("login")).submit();
        eyes.checkWindow("Logged In");
        Assert.assertTrue("success message should be present after logging in",
                driver.findElement(By.cssSelector(".flash.success")).isDisplayed());
        eyes.close();
    }
...

With eyes.checkWindow(); we are specifying when in the test’s workflow we’d like Applitools Eyes to capture a screenshot (along with some description text). For this test we want to check the page before logging in, and then the screen just after logging in — so we use eyes.checkWindow(); two times.

NOTE: These visual checks are effectively doing the same work as the pre-existing assertion (e.g., where we’re asking Selenium if a success notification is displayed and asserting on the Boolean result) — in addition to reviewing other visual aspects of the page. So once we verify that our test is working correctly we can remove this assertion and still be covered.

We end the test with eyes.close. You may feel the urge to place this in teardown, but in addition to closing the session with Eyes, it acts like an assertion. If Eyes finds a failure in the app (or if a baseline image approval is required), then eyes.close will throw an exception; failing the test. So it’s best suited to live in the test.

NOTE: An exceptions from eyes.close will include a URL to the Applitools Eyes job in your test output. The job will include screenshots from each test step and enable you to play back the keystrokes and mouse movements from your Selenium tests.

When an exception gets thrown by eyes.close, the Eyes session will close. But if an exception occurs before eyes.close can fire, the session will remain open. To handle that, we’ll need to add an additional command to our teardown.

// filename: Login.java
...
    @After
    public void teardown() {
        eyes.abortIfNotClosed();
        driver.quit();
    }
}

eyes.abortIfNotClosed(); will make sure the Eyes session terminates properly regardless of what happens in the test.

Now when we run the test, it will execute locally while also performing visual checks in Applitools Eyes.

What About Other Browsers?

If we want to run our test with it’s newly added visual checks against other browsers and operating systems, it’s simple enough to add in Sauce Labs support.

NOTE: If you don’t already have a Sauce Labs account, sign up for a free trial account here.

First we’ll need to import the relevant classes.

// filename: Login.java
...
import org.openqa.selenium.Platform;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
...

We’ll then need to modify the test setup to load a Sauce browser instance (via Selenium Remote) instead of a local Firefox one.

// filename: Login.java
...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", "Login succeeded");
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                "YOUR_SAUCE_USERNAME",
                "YOUR_SAUCE_ACCESS_KEY");
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        eyes = new Eyes();
        eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));
        driver = eyes.open(browser, "the-internet", "Login succeeded");
    }
...

We tell Sauce what we want in our test instance through DesiredCapabilities. The main things we want to specify are the browser, browser version, operating system (OS), and name of the test. You can see a full list of the available browser and OS combinations here.

In order to connect to Sauce, we need to provide an account username and access key. The access key can be found on your account page. These values get concatenated into a URL that points to Sauce’s on-demand Grid.

Once we have the DesiredCapabilities and concatenated URL, we create a Selenium Remote instance with them and store it in a local browser variable. Just like in our previous example, we feedbrowser to eyes.open and store the return object in the driver variable.

Now when we run this test, it will execute against Internet Explorer 8 on Windows XP. You can see the test while it’s running in your Sauce Labs account dashboard. And you can see the images captured on your Applitools account dashboard.

A Small Bit of Cleanup

Both Applitools and Sauce Labs require you to specify a test name. Up until now, we’ve been hard-coding a value. Let’s change it so it gets set automatically.

We can do this by leveraging a JUnit TestWatcher and a public variable.

// filename: Login.java
...
import org.junit.rules.TestRule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;
    public String testName;

    @Rule
    public TestRule watcher = new TestWatcher() {
        protected void starting(Description description) {
            testName = description.getDisplayName();
        }
    };
...

Each time a test starts, the TestWatcher starting function will grab the display name of the test and store it in the testName variable.

Let’s clean up our setup to use this variable instead of a hard-coded value.

// filename: Login.java
...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", testName);
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                System.getenv("SAUCE_USERNAME"),
                System.getenv("SAUCE_ACCESS_KEY"));
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        eyes = new Eyes();
        eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));
        driver = eyes.open(browser, "the-internet", testName);
    }
...

Now when we run our test, the name will automatically appear. This will come in handy with additional tests.

One More Thing

When a job fails in Applitools Eyes, it automatically returns a URL for it in the test output. It would be nice if we could also get the Sauce Labs job URL in the output. So let’s add it.

First, we’ll need a public variable to store the session ID of the Selenium job.

// filename: Login.java
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;
    public String testName;
    public String sessionId;
...

Next we’ll add an additional function to TestWatcher that will trigger when there’s a failure. In it, we’ll display the Sauce job URL in standard output.

// filename: Login.java
...
    @Rule
    public TestRule watcher = new TestWatcher() {
        protected void starting(Description description) {
            testName = description.getDisplayName();
        }

        @Override
        protected void failed(Throwable e, Description description) {
            System.out.println(String.format("https://saucelabs.com/tests/%s", sessionId));
        }
    };
...

Lastly, we’ll grab the session ID from the Sauce browser instance just after it’s created.

// filename: Login.java
...
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        sessionId = ((RemoteWebDriver) browser).getSessionId().toString();
...

Now when we run our test, if there’s a Selenium failure, a URL to the Sauce job will be returned in the test output.

Expected Outcome

  • Connect to Applitools Eyes
  • Load an instance of Selenium in Sauce Labs
  • Run the test, performing visual checks at specified points
  • Close the Applitools session
  • Close the Sauce Labs session
  • Return a URL to a failed job in either Applitools Eyes or Sauce Labs

Outro

Happy Testing!

 

About Dave Haeffner: Dave is the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by hundreds of testing professionals) as well as a new book, The Selenium Guidebook. He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.

Sauce Labs Appoints Technology Veteran Charles Ramsey as Company’s First Chief Revenue Officer

February 24th, 2015 by Amber Kaplan

SAN FRANCISCO, CA-  Sauce Labs, Inc., the leading cloud-based web and mobile application testing platform, today announced that it has appointed Charles Ramsey as the company’s first Chief Revenue Officer (CRO). Ramsey brings more than 25 years of industry experience and insight to his role at Sauce Labs. He will be responsible for all customer-facing areas, including sales, business development, and marketing, as well as continuing to build on Sauce Labs record 2014 results as the company extends its leadership position in the booming automated testing market.

“Charles is an innovative strategist, committed to building strong relationships with customers and partners,” said Jim Cerna, CEO of Sauce Labs. “His demonstrated ability to strategically grow companies will help us address the exploding demand we are seeing in the market for our technology. With Charles’ experience, leadership, and track record of repeated successes, we are poised to continue our rapid growth trajectory through 2015 and beyond.”

“Sauce Labs is well-positioned to take advantage of the dramatic proliferation of web and mobile apps across a variety of devices and operating systems. I look forward to working with the entire Sauce team to further the company’s market leadership and growth by bringing new levels of innovation, customer experience, and value to the marketplace,” said Ramsey.

Sauce Labs provides an instantly scalable testing cloud that is optimized for continuous integration (CI) and continuous delivery (CD). When tests are automated and run in parallel on virtual machines across multiple browsers and platforms, testing time is reduced and developer time is freed up from managing infrastructure. When paired with a CI system, developers can easily test web, hybrid and native applications early on in their development cycles, continuously and affordably. Sauce Labs currently supports more than 480 browser, operating system and device platform combinations.

Prior to joining Sauce Labs, Ramsey was an early member of the Quest Software management team, where he served as vice president of World Wide Marketing and Sales. He is a former Venture Partner at JMI Equity and has also served on the Board of Directors at notable companies such as Configuresoft, Inc. and ServiceNow, Inc. Early in his career, Ramsey rose to vice president, North America Sales for Computer Intelligence, a division of Ziff Davis, after beginning his career with the IBM National Accounts Division in a variety of sales assignments. He has a Bachelor of Arts from the University of California, San Diego and a Master of Information Management from the American Graduate School of International Management.

Helpful Links

About Sauce Labs
Sauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity and reduce infrastructure costs for software teams of all sizes.

Sauce Labs is a privately-held company funded by Toba Capital, Salesforce Ventures, Triage Ventures and the Contrarian Group. For more information, please visit http://saucelabs.com.

Automated Visual Testing In The Cloud: Enhance Your Cross-browser Coverage [RECAP]

February 19th, 2015 by Bill McGee

Thanks for joining us for our last webinar, Automated Visual Testing In The Cloud: Enhance Your Cross-browser Coverage, featuring Adam Carmi from Applitools and Chris Broesamle from Sauce Labs.

In this webinar, Adam and Chris explained how to avoid visual regressions and front-end bugs by adding scalable automated visual testing to existing Selenium and Appium tests and running them on the Sauce Labs cloud. They also:

  • Revealed expert tips on how to successfully perform large-scale automated visual testing
  • Showed how to leverage visual testing to increase coverage, while reducing maintenance efforts
  • Ran a live cross-browser visual test with Sauce Labs and Applitools

Missed the presentation, want to hear it again, or share with a colleague?

Listen to the recording HERE and view the slides below.

Continuous Testing in Practice [WEBINAR]

February 18th, 2015 by Amber Kaplan

As web and mobile application software increases in complexity, the number and frequency of tests grows exponentially. But managing your tests with sub-optimal continuous integration (CI) workflows can lead to bottlenecks, delays, and lost developer productivity.

In our next webinar, Continuous Testing in Practice, Ophir Prusak from BlazeMeter and Abhijit Pendyal from Sauce Labs will show you how to integrate automated testing into your CI process so that you can test early and often to speed up deployment.

Ophir and Abhijit will cover:

  • Why Continuous Testing is so important today
  • How to ensure testing keeps pace with agile development cycles
  • The end-to-end flow of a continuous testing process
  • How to implement continuous automated functional & performance testing
  • How to integrate continuous testing with your existing tools

Join us for this presentation on Tuesday, February 24th at 10am PST/1pm EST. There will be a Q&A with both Ophir and Abhijit following the end of the presentation.

Click HERE to register today.

Continuous Testing In Practice

February 12th, 2015 by Amber Kaplan

Cloud-based load testing service BlazeMeter released a white paper that discussed continuous testing last week. The ‘Continuous Testing’ platform leverages cloud and open source technology to bring better testing for DevOps and Agile practices enabling shorter release cycles with higher quality releases.

You might not normally consider back-end load testing as part of a continuous delivery process, but we believe in testing all the things. By pairing BlazeMeter’s service with Sauce Labs’ readily available test cloud used for functional and unit testing and others such as performance testing with New Relic, users get access to a full spectrum of tests that help ensure the utmost application quality- and that’s a big win for everyone.

To learn more about this continuous testing solution:

For more information about BlazeMeter’s newest offering, see below – or read the full press release here.

Read the rest of this entry »

New Web and Mobile Testing Trends Research Report

February 11th, 2015 by Bill McGee

Sauce Labs recently partnered with Dimensional Research to conduct an independent survey of technology professionals to understand current trends in web and mobile testing. With the proliferation of devices, browsers and operating systems, and the rapid adoption of modern software development methodologies, we wanted to see how development teams are balancing these complex user environments with current testing resources and the impact agile methodologies have had on software testing. Perhaps not surprisingly, 94% of respondents who have adopted continuous integration (CI)  and other agile methodologies said they’ve changed their approach to testing, and for the better – they’re testing earlier and more often, and are able to do more of all types of testing. You can see the full report here – happy reading!

SauceLabs-Infographic-v3

 

 

Automated Visual Testing In The Cloud: Enhance Your Cross-browser Coverage [WEBINAR]

February 10th, 2015 by Amber Kaplan

Manually testing your app for UI bugs across multiple browsers and devices is a tedious process, but you can accelerate it with automated visual testing that goes beyond pixel-by-pixel screenshot comparison.

In this webinar, you’ll see how you can easily avoid front-end bugs and visual regressions, as well as substantially increase coverage, by adding scalable automated visual testing to your existing Selenium and Appium tests running on the Sauce Labs cloud.

In this webinar we’ll:

  • Reveal expert tips on how to successfully perform large-scale automated visual testing
  • Show how to leverage visual testing to increase coverage, while reducing maintenance efforts
  • Run a live cross-browser visual test with Sauce Labs and Applitools

Join us for this presentation on Tuesday, February 17th at 11am PST. There will be a Q&A with both Applitools and Sauce Labs following the end of the presentation.

Click HERE to register today.

Stop Being A Language Snob: Debunking The ‘But Our Application Is Written In X’ Myth [Guest Post]

February 6th, 2015 by Adam Goucher

If there is one myth in the [browser] automation world that drives me crazy it is that browser automation scripts need to be written in the same language as the application is written in. It seems like that should be a Good Idea; in principle, but in reality it is actually responsible for a lot of ‘failed’ automation efforts.

Let’s choose a language to pick on. How about C# using ASP MVC; has a large user base (especially in the enterprise space) and pretty mature stack to use. (We could have picked any language…)

So now we have a nice ASP MVC application that we think is going to solve some customer’s burning needs and of course it’s nicely unit tested because you are doing some variant of TDD/BDD. Your browser automation scripts should naturally be written in C#, right?

No.

Well, actually, ‘maybe’. Read the rest of this entry »

Sauce Labs Secures $15 Million For Geographic And Infrastructure Expansion

February 5th, 2015 by Amber Kaplan

SAN FRANCISCO – February 5, 2015 Sauce Labs, Inc., the leading cloud-based web and mobile application testing platform, today announced it has secured an additional $15 million in Series D expansion funding from current investor, Toba Capital. The news comes as Sauce Labs experiences overwhelming growth with over 200 million total tests run on its platform, and 145 percent revenue growth in 2014.

The new round of funding will extend both the company’s core web application testing offering and its mobile testing platform, as well as provide continued backing of Appium, the premier open source mobile automation tool developed by Sauce Labs and a thriving community of open source contributors. Additionally, it will be used to expand the company’s infrastructure, development team, and international operations.

“Automated testing is critical to modern software development,” said Jim Cerna, CEO of Sauce Labs. “We’ve seen tremendous adoption among web developers, and as mobile developers refine their continuous integration and continuous delivery processes, there is growing demand for scalable, reliable testing for mobile apps, as well. This new round of funding will help us continue to build out the infrastructure to meet this demand.”

“As software teams embrace continuous integration and delivery processes, automated testing plays a crucial role in improving quality and time to market, for both web and mobile applications. Enterprises and organizations of all sizes turn to us because we help them make their delivery pipelines efficient and reliable, while improving software quality along the way,” said Steve Hazel, co-founder and chief product officer of Sauce Labs.

Sauce Labs provides an instantly scalable testing cloud that is optimized for continuous integration (CI) and continuous delivery (CD). When tests are automated and run in parallel on multiple virtual machines across multiple browsers and platforms, testing time is reduced and developer time is freed up from managing infrastructure. When paired with a CI system, developers can easily test web, hybrid and native applications early on in their development cycles, continuously and affordably. Sauce Labs currently supports more than 450 browser, operating system and device platform combinations.

“The application development market is rapidly shifting with the adoption of continuous integration (CI) and continuous delivery (CD),” said Tyler Jewell, partner at Toba Capital and Sauce Labs board member. “CI and CD eliminate the time constraints that inhibit teams from achieving their full market potential, making development, support and release instantaneous and ongoing. We are thrilled to support Sauce Labs as they continue to make these CI/CD processes more achievable and effective for every software development team.”

Helpful Links

 

 

About Sauce Labs
Sauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity and reduce infrastructure costs for software teams of all sizes.

Sauce Labs is a privately-held company funded by Toba Capital, Salesforce Ventures, Triage Ventures and the Contrarian Group. For more information, please visit http://saucelabs.com.

DZone: Guide To Continuous Delivery [DOWNLOAD]

February 4th, 2015 by Amber Kaplan

DZone 2015 Guide to Continuous Delivery

We are very excited to be a research partner for DZone’s Guide to Continuous Delivery, a premium resource focused on continuous integration and DevOps management trends, strategies, and tools.

Readers of the guide will get an overview of continuous delivery practices and how continuous delivery affects many aspects of an organization. This guide includes:

  • Articles written by continuous integration and deployment experts – including Sauce labs
  • Detailed profiles of 35+ continuous delivery tools and solutions
  • “Continuous Delivery Maturity Checklist” that gauges where your continuous delivery skills rank
  • “Continuous Delivery: Visualized” infographic that details the tools developers use at every stage of the pipeline

DZone’s continuous delivery guide offers key insights into continuous integration and through a survey of 750+ developers and experts, allowing readers to learn trends from practitioners in the technology professional community. Additionally, the guide’s solutions directory compares various tools for continuous integration, application release automation, and configuration management to help readers wisely choose the solutions they need.

Download a free copy of the guide HERE.

About DZone

DZone provides expert research and learning communities for developers, tech professionals, and smart people everywhere.  DZone has been a trusted, global source of content for over 15 years.