Decreasing False Positives in Automated Testing [RECAP]

March 30th, 2015 by Amber Kaplan

Thanks for joining us for our last webinar, Decreasing False Positives in Automated Testing, featuring Anand Ramakrishnan of QASource.

In this webinar, Anand covered: Read the rest of this entry »

Appium Version 1.3.7 Released on Sauce Labs

March 26th, 2015 by Eric Millin

Appium logo w- tagline {final}-01

We’re pleased to announce that Appium version 1.3.7 is available on Sauce. This small release includes two hotfixes:

General

  • fixed a failure to remap session id in proxied responses

iOS

  • fixed intermittent failure to find Xcode

Repost: Testing in a Real Browser with Sauce Labs + Travis CI

March 26th, 2015 by Amber Kaplan

This post comes from our friend Sam Saccone, who wrote a nice how-to on using Sauce Connect and Travis CI. Check out the original post on his blog.

I recently found myself implementing a basic set of JavaScript integration tests on an open source project. I quickly came to the realization that there is a serious lack of good documentation on how to get a basic browser test running on a Travis CI using Sauce Labs and Sauce Connect.

After stumbling across the barren desert of outdated doc pages, incorrect stackoverflow answers, and ancient google group posting, I present to you the spoils of my quest to the underbelly of the web.

Let’s approach this in the context of a real problem: Testing javascript in a real web browser. Read the rest of this entry »

Jason Huggins: Fixing HealthCare.gov, One Test at a Time [MEETUP]

March 24th, 2015 by Amber Kaplan

jason-hugginsYou may recall that our co-founder Jason Huggins took a leave of absence to help fix HealthCare.gov. He’ll be back on April 21 to talk about his experiences there at the next Selenium Meetup in San Francisco. RSVP before it fills up! Info below.

Date + Time: Tuesday, April 21, 2015 at 6:30 PM

Location: If (we): 848 Battery Street, San Francisco, CA Read the rest of this entry »

Repost: Lessons Learned from Automating iOS Apps – What To Do When Tests Require Camera Roll Resources?

March 18th, 2015 by Amber Kaplan

This post comes from our friend Jay Sirju at Animoto, who is leveraging Sauce, Appium, and CI methodologies to automate mobile testing for their iOS application.  Check out the original post on their blog.

Here at Animoto, the mobile application development team had spent some time over the past year investigating and implementing CI methodologies into the development cycle of the Animoto Video Maker application for iOS. A major part of this initiative involved creating automated test cases that would run at various times and circumstances.

First a bit of background. When we started this, we had already implemented a good amount of automation for the Animoto website. We had chosen to use Selenium and ran our automated tests against various browsers using Sauce Labs. We decided to extend our existing infrastructure to support running automated tests using theAppium library against Sauce Labs.   For those unfamiliar with mobile testing on Sauce Labs, they use the iOS and Android Simulators to run tests. I know, not ideal, but we can get to that another time.

The Problem

For anyone who has ever launched a fresh iOS Simulator (before Xcode 6), the OS is in it’s factory state. The Animoto Video Maker App transforms your pictures, video, and text, into professional looking videos… see where I’m going?

Screen-Shot-2015-02-18-at-11.31.04-AM

A lot of user flows depend on having some photos in the camera roll. A factory-fresh simulator without any photos means there are man flows we can’t automate. Unfortunately, at the time of writing, Sauce Labs does not have a way to upload assets to populate the Camera Roll, and the simulators are reset after executing each test case. Starting with XCode 6, the Camera Roll does have some images out of the box, but what was really needed were meaningful pictures and videos. So what is needed is a way to populate the Camera Roll while working within the constraints of running tests in Sauce Labs. Well, the app already reads images and pictures from Camera Roll, what about writing to it as well?

Adding or Altering Configuration Profile

Before we get to actually populating the Camera Roll, we need a mechanism to ensure that this logic is performed only when the intent is to run automation. Out of the box, Xcode provides 3 build configurations (Debug, Release, and Distribution). We can edit these configurations, add new ones, or even delete unnecessary ones. In this case, we can simply add a Test configuration to the mix. Once we did that, we were able to change various build settings to help create hooks for app automation. We can start out by adding a Preprocessor Macro for the Test build configuration, so that we can tell the pre-processor when to compile test hooks into the build.

animoto 2

Okay, now we can do some fun stuff with the build. For the sake of brevity, let’s focus specifically on the original issue: Getting pictures and videos into the Camera Roll.

Change Test Configuration Profile Settings

First things first – how does one get pictures and videos up to Sauce Labs? We can simply add them to the iOS project, but that would increase the size of the application bundle regardless of which build configuration is being used. Definitely not ideal. A better choice would be to store them somewhere externally and copy them to the application bundle when the Test configuration is used. This can be done by running a script when building the project.

if [ ${CONFIGURATION} == "Test" ]; then
cp -r ${PICTURE_AND_VIDEO_LOCATION}/ ${BUILT_PRODUCTS_DIR}/${PRODUCT_NAME}.app
fi

Populating the Camera Roll

Now we have an application bundle that contains a bunch of sample pictures and videos. This is great because when the application gets uploaded to Sauce Labs for testing, so do all the sample data.   The following code example assumes all the sample images are in a folder within the application bundle named ‘TestImages’:

+ (void) populateCameraRoll
{
    NSString* absolutePicturePath = [[NSBundle mainBundle] pathForResource:@"TestImages" ofType:nil];
    NSArray* pictureList = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:absolutePicturePath error:nil];

    for(int i = 0; i < [pictureList count]; i++)
    {
        NSString* absolutePictureFilePath =[ NSString stringWithFormat:@"/%@/%@", absolutePicturePath,[pictureList objectAtIndex: i]];

        NSData *jpeg = [NSData dataWithContentsOfFile:absolutePictureFilePath];

        UIImage *image = [UIImage imageNamed:absolutePictureFilePath];

        CGImageSourceRef source = CGImageSourceCreateWithData((__bridge CFDataRef)jpeg, NULL);
        CFDictionaryRef imageMetaDataRef = CGImageSourceCopyPropertiesAtIndex(source,0,NULL);
        NSDictionary *imageMetadata = CFBridgingRelease(imageMetaDataRef);
        CFRelease(source);

        if (image != nil)
        {
            ALAssetsLibrary* library = [[ALAssetsLibrary alloc] init];

            dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
                dispatch_semaphore_t sema =  dispatch_semaphore_create(0);

                [library writeImageToSavedPhotosAlbum:[image CGImage] metadata:imageMetadata completionBlock:^(NSURL *assetURL, NSError *error)
                 {
                     dispatch_semaphore_signal(sema);
                 }];
                dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);
            });
        }
    }
}
@end

The above code makes a bunch of assumptions. First, it only works for images. Another folder consisting of sample videos can be created, and the using similar logic and the writeImageToSavedPhotosAlbum method. Next, calling writeImageToSavedPhotosAlbum can indeed fail. For the sake of keeping this example code more readable, error handling was excluded. Retry logic should be included if an error is returned.

Finally, you may have noticed the use of a semaphore in the example code. Writing images to the Camera Roll is actually an asynchronous call, meaning that the call returns immediately while a separate thread processes writing the image data to the Camera Roll. The writeImageToSavedPhotosAlbum method can fail if there are too many threads trying to write image data simultaneously. The semaphore is used ensure that images are written to the Camera Roll sequentially. This makes using the writeImageToSavedPhotosAlbum method much more stable.

Okay, so now that is left is to call the method when running the Test configuration. This can easily be done using the Preprocessor Macro setting that was above mentioned.

#if TEST
[ANTestClass populateCameraRoll];
#endif

It is recommended to call the method somewhere deterministic (ie -a button tap). Simply populating the Camera Roll at start-up may mess up launching Apples Instrumentation library because of the Alert displayed when the app accesses the Camera Roll for the first time. This lesson was learned the hard way.

This opens up the ability to add more automated test hooks into the application under test, but as a word of warning; the more hooks added, the more the test configuration of the app diverges from what is being released to customers.

 

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Decreasing False Positives in Automated Testing [WEBINAR]

March 17th, 2015 by Amber Kaplan

False positives: automated testing’s arch nemesis.

When automated tests are written well, they are part of a healthy CI/CD process that can save developer time and company money. But when a team gets false positives from unreliable tests, the entire build can get derailed. What’s worse, too many false positives can erode an organization’s belief in the value of using a test automation framework at all.

In this webinar, Anand Ramakrishnan of QASource will walk you through what false positives are and provide key strategies to reduce or eliminate them.

Anand will cover:

• What false positives are and why they occur

• Common causes and the challenges they create

• How to implement key strategies to reduce them

• Why implementing these strategies is essential for increasing productivity and reducing cost, as well as time to market

• Case studies, real world examples, a live demo and Q&A

Join us for this presentation on Tuesday, March 24th at 11am PDT/2pm EDT. There will be a Q&A following the end of the presentation.

Click HERE to register today.

Repost: Angular + Protractor + Sauce Connect, Launched From Gulp, All Behind A Corporate Firewall!

March 9th, 2015 by Amber Kaplan

This post comes from our friend Stephen Wylie, who is using Sauce Connect to work with his corporate firewall.  Check out the original post on his blog.

You didn’t think it could be done, did you?

Well, let me prove you wrong!  First, some terms:
  • AngularJS: An MVC framework for JavaScript, allowing you to write web apps without relying on JQuery.
  • Protractor: A test harness for Angular apps.
  • Sauce Labs: A company providing cloud services to help you run E2E testing on your web app in any environment combination.
  • Node.js: Package manager for JavaScript.  You’ll need this for installing all the dependencies to get this tool set working.
  • Gulp: A build manager and mundane-task automator.  Competitor with the older, well-entrenched Grunt, but gaining popularity by the hour.  Uses JavaScript syntax, but could theoretically be used as a Makefile or shell script replacement.

The Basic Premise

My organization is writing web apps using Angular.  Long before I joined, they selected Gulp to manage application tasks such as allowing it to run on localhost at port 8888 for development & unit test purposes.  They also selected Protractor as a test harness to interact with the web app.  Protractor depends on the presence of Angular in order to work properly, and provides the use of Selenium WebDriver (for interacting with browsers) and unchained promises (a JavaScript construct to avoid callback functions).

 

Sauce Labs has been selected as the testing tool of choice because it saves us from having to set aside a massive amount of infrastructure to run tests on multiple platforms.  Through the configuration file for Protractor, I can specify exactly what OS platform & browser combination I want the test to run on.  Of course, being an organization such as it is, they also have a corporate firewall in place that will prevent the VMs at Sauce Labs from accessing development & test deployments of our web apps under construction under normal circumstances.  This is where Sauce Connect comes in: it provides a secure mechanism for the external Sauce Labs VMs to acquire the data that the server would serve to you as if you were inside the corporate firewall.  Winful for everybody!  The best part is that Sauce Labs is free for open-source projects.

Journey Through the Forest: Wiring All This Together

It is, truthfully, “stupid simple” to set up a Gulp task that will run Protractor tests through the Sauce Connect mechanism.  All you need in your Protractor configuration file is:
exports.config = {
    sauceUser: "your login name",
    sauceKey: "the GUID provided to you on your dashboard on Sauce's site",
    specs: ["the files to run as your test"],
    sauceSeleniumAddress: "this is optional: default is ondemand.saucelabs.com:80/wd/hub, but localhost:4445/wd/hub is also valid for when you're running sc locally and ondemand doesn't work",
    capabilities: {
        'tunnel-identifier': 'I will explain this later',
        'browserName': "enter your browser of choice here"
    }
}

(Note that where it says “:4445″ above should be replaced by the port number specified by the sc binary if it says anything different.) It’s so simple that you don’t even need any “require()”s in the config file. And in your Gulpfile, all you need is this:

gulp.task('sauce-test', function() {
    gulp.src('same as your "specs" from above, for the most part (unless your working directory is different)')
    .pipe((protractor({
        configFile: 'path to the config file I described above'
    })).on('error', function (e) {
        throw e;
    }).on('end', function() {
        // anything you want to run after the Sauce tests finish
    }));
});

Then, of course, you can run your tests by writing “gulp sauce-test” on the command line set to the same directory as the Gulpfile.  However, proper functioning of this configuration eluded me for a long time because I did not know the Sauce Connect binary (“sc” / “sc.exe”)was supposed to be running on my machine.  I thought the binary was running on another machine in the organization, or on ondemand.saucelabs.com, and all I needed to do was set the settings in the Gulpfile to the instance of sc that’s remote (with the SauceSeleniumAddress entry).  While I could point the SauceSeleniumAddress to a different host, it was a flawed assumption on my part that anyone else in my organization was running “sc” already.  Also, ondemand.saucelabs.com might not answer the problem because it doesn’t provide the services in “sc” by itself.  It is most convenient to run sc on your own system.

This configuration issue stymied me so much that I actually played with Grunt and several plugins therein before realizing that running tests through Sauce Connect was even possible through JavaScript to any extent.  Ultimately, I found a Node plugin for Grunt called “grunt-mocha-webdriver” that proved to me this was possible, and even doable in Gulp with Protractor and Selenium-WebDriver like I want, as opposed to Grunt/Mocha/WD.js.  (By the way, blessings to jmreidy, since he also wrote the sauce-tunnel which is relied upon heavily in this tutorial.)

Nevertheless, the easiest way to run Sauce Connect on your own system is to install the “sauce-tunnel” package through npm, the Node Package Manager (visit https://www.npmjs.com/ for other hilarious things “npm” could stand for :-P).  This is, of course, achievable by running the following on the command line:

npm install sauce-tunnel

If sauce-tunnel is already in your node_modules directory, then good for you!  Otherwise, you could run this in any directory that “npm” is recognized as a valid command, but you might want to place this module strategically; the best place to put it will be revealed below.  Nevertheless, you need to traverse to the directory where sc is located; this depends on what OS you are running, as the sauce-connect package contains binaries for Mac OSX (Darwin), Linux 32/64-bit, and Windows.  So, run the “sc” executable for your given platform before you run the Gulp task specified above, or else Gulp will appear to time out (ETIMEDOUT) when it’s trying to get into Sauce Connect.

 

The minimum options you need for sc are your Sauce login name and your Sauce key (the GUID as specified above).  There are more options you can include, such as proxy configurations, as specified in the Sauce Connect documentation.  (Note that the tunnel-identifier, as called out in the Protractor config file, can be specified as an argument to sc.)

In simple terms, here’s what we have thus far:

[assuming you’ve set up all the Node packages]:

vendor/[platform]/bin$ sc -u -k [-i ] [other options]

gulp-workingdir$ gulp sauce-test

This will set up “sc” for as long as your computer is hooked up to the Internet, and will run the Sauce tests on the existing tunnel.  The tunnel will remain active until you disconnect your computer from the Internet or end the sc process, but the tests running through Gulp will set up & tear down a Selenium WebDriver that’ll drive the UI on your web app.

Help!  The test did not see a new command for 90 seconds, and is timing out!!!

If you are seeing this message, you might be behind a corporate proxy that is not letting your request go straight through to the Sauce servers.  Protractor has in its “runner.js” file a section where it will pick a specific DriverProvider based on certain settings you provide in the configuration file, and by providing the “sauceUser” and “sauceKey” values, it will pick the “sauce” DriverProvider.  The sauce DriverProvider provides an “updateJob” function that communicates with Sauce Labs (via an HTTP PUT request) on the status of the job.  This function is supposed to run after the tests conclude, and if that HTTP request fails, then the Gulp task will not end properly; thus, you will see this message.  Your list of tests in your Sauce Connect dashboard will look like this:
Screen Shot 2015-03-05 at 7.06.50 PM
This message is so severe in Sauce that it doesn’t just show up as “Fail,” it shows up as “Error”.  It also wastes a bunch of execution time, as seen in the picture above, and will obscure the fact that all the test cases actually passed (as they did in the pictured case above).  If you see this message after it is apparent that there are no more commands to be run as part of the test, then it is probably a proxy issue which is easy to resolve.

 

Here’s how:

 

In your Protractor configuration file, add the following lines:
var HttpsProxyAgent = require("https-proxy-agent");

var agent = new HttpsProxyAgent('http://<user>:<password>@<proxy host>:<port>');

exports.config = {
    agent: agent,
    // things you had in there before
};

Then, in your node_modules/protractor/lib/driverProviders/sauce.js file (i.e. the DriverProvider for Sauce Labs in Protractor), add this:

this.sauceServer_ = new SauceLabs({
    username: this.config_.sauceUser,
    password: this.config_.sauceKey,
    agent: this.config_.agent    // this is the line you add
});

Once you have your https-proxy-agent in place as specified, your PUT request should go through, and your tests should pass (as seen in the Sane jobs).

The whole process, end-to-end, running in Gulp

If it does not satisfy you to simply run the “sc” binary from the command line and then kick off a Gulp task that relies on the tunnel already existing, you can get everything to run in Gulp from end to end. To do this, you need to require sauce-tunnel in your Gulpfile (thus you might as well run npm install sauce-tunnel from the same directory that your Gulpfile exists). Then, you need to make some changes to the Gulpfile: add some additional tasks for tunnel setup & teardown, and some special provisions so these tasks are executed in series rather than in parallel.

var SauceTunnel = require('sauce-tunnel');
var tunnel;

gulp.task('sauce-start', function(cb) {
    tunnel = new SauceTunnel("<your Sauce ID>", "<Your Sauce Key>", "<Sauce tunnel name -- this must be specified and match the tunnel-identifier name specified in the Protractor conf file>");
    // >>>> Enhance logging - this function was adapted from that Node plugin for Grunt, which runs grunt-mocha-wd.js
    var methods = ['write', 'writeln', 'error', 'ok', 'debug'];
    methods.forEach(function (method) {
        tunnel.on('log:'+method, function (text) {
            console.log(method + ": " + text);
        });
        tunnel.on('verbose:'+method, function (text) {
            console.log(method + ": " + text);
        });
    });
    // <<<< End enhance logging

    tunnel.start(function(isCreated) {
        if (!isCreated) {
            cb('Failed to create Sauce tunnel.');
        }
        console.log("Connected to Sauce Labs.");
        cb();
    });
});

gulp.task('sauce-end', function(cb) {
    tunnel.stop(function() {
        cb();
    });
});

gulp.task('sauce-test', ['sauce-start'], function () {
    gulp.src('<path to your Protractor spec file(s)>')
    .pipe((protractor({
        configFile: '<path to your Protractor conf file>'
    })).on('error', function (e) {
        throw e;
    }).on('end', function() {
        console.log('Stopping the server.');
        gulp.run('sauce-end');
    }));
});

Note here that the cb() function is new to Gulp, yet the “gulp.run()” construct mentioned toward the bottom of the code snippet above is actually deprecated. I will get around to fixing that once it stops working, but I think that in the grand scheme of priorities, I’d rather clean the second-story gutter with only a plastic Spork first before fixing that deprecated line. :-P

At this point, you should be able to run a test with Sauce Connect from end to end in Gulp without any extra intervention. However, if Gulp is failing because it can’t write to a file in a temporary folder pertaining to the tunnel (whose name you picked), then you can always run gulp as root find a way to have it save to a different temporary location that you have access to, since it’s always good to minimize running things as root.

One Brief Important Interruption about Lingering sc Instances…

If these instructions haven’t worked out 100% for you, or you are me and spent a great deal of time exploring this, you may be frustrated with how many times Sauce Connect hangs around when there’s been a problem. You can’t start the Sauce Connect binary again if it’s already running, yet if you try to do this, it gives you an esoteric error message that does not make it apparent that this is indeed what happened. To remedy this in a *nix operating system, simply write “pkill sc”, as long as you don’t have other critical processes that have “sc” in their name. In my case, the other processes with “sc” in the name are running under a different user, and I don’t have privileges to kill them (I’m not logged in as root nor running “sudo pkill sc”), so it doesn’t do anything harmful to the system.

Shutting It Down Cleanly

In order to properly shut down sc, you may have noticed one final Gulp task in the code snippet above — “sauce-end”. This task, in the background, runs an HTTP DELETE operation on saucelabs.com, and is subject to corporate proxy rules once again. To circumvent this, you can simply require https-proxy-agent in node_modules/sauce-tunnel/index.js (like we did in the Protractor configuration file), and set up the agent in the same way. In this case, you will edit the code in node_modules/sauce-tunnel/index.js as such:

// other pre-existing requires
var HttpsProxyAgent = require("https-proxy-agent");

var agent = new HttpsProxyAgent('http://<user>:<password>@<proxy host>:<port>');

// other existing code
this.emit('verbose:debug', 'Trying to kill tunnel');
request({
  method: "DELETE",
  url: this.baseUrl + "/tunnels/" + this.id,
  json: true,
  agent: agent    // this is the line you add
}, // ... etc

Now, obviously, this is not sustainable if you wish to ever upgrade sauce-tunnel or wish not to include a proxy agent. For this, I will be submitting “less hacky” fixes to the respective GitHub repositories for these open-source Node modules in order to make it easier for all users in the future to use Sauce Connect with Protractor through their corporate proxies.

Nevertheless, there’s no harm in this DELETE call failing, other than it makes the Gulp task stall another minute or so, which is annoying when you’re at work late trying to learn how all this stuff works in order to finish off some polishing touches on your big project.

To recap running everything from end to end in Gulp:

[Assuming you’ve set up all your Node packages to run a Protractor script with the conf file set up for Sauce Labs, as described above]:

  • In the same directory as your Gulpfile, run:
    npm install sauce-tunnel
  • Set up your Gulpfile in the manner I described above, with the sauce-tunnel require, and the “sauce-start”, “sauce-end”, and “sauce-test” tasks, and with the “Sauce tunnel name” (3rd argument in new SauceTunnel()) set to the same value as the Protractor config file “tunnel-identifier” value. Be sure to study all the possible values that “new SauceTunnel()” takes, as you can pass in options to the sc binary if you need them.
  • If you are behind a corporate proxy or firewall, make the recommended edits to the Sauce DriverProvider at node_modules/protractor/lib/driverProviders/sauce.js, and to the sauce-tunnel module at node_modules/sauce-tunnel/index.js.
  • Run the Gulp task.
    gulp sauce-test
    or
    sudo gulp sauce-test

Once again, I plan to check in “more sustainable” and “less hacky” code to help you deal with corporate proxies in the future without making temporary workarounds to downloaded modules.

 

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Appium 1.3.6 Available on Sauce

March 4th, 2015 by Eric Millin

Appium logo w- tagline {final}-01The Appium team is pleased to announce that version 1.3.6 is now available on Sauce.  The new build includes changes from 1.3.5, which was previously unavailable for Sauce users.

iOS
– fix for a bug when driver.get() never returns for page with alert.
– iOS 8.2 support.
– fixed safari startup crashes.
– ensure Appium drops into the right continuation cb when selecting hybrid contexts.

Android
– fix XPath regression where Appium failed to recognize non-ASCII characters.
– fix regression where Appium failed to set ADB’s path during Chromedriver tests.
– now finds the location of adb earlier.
– ensure encoding stream in Bootstrap.jar closes correctly.
– add workaround for issue where UiAUtomator fails to find visible elements.
– fixed undefined member error for the release object.
– add a delete key test.

Selendroid
– upgrade to Selendroid 0.13.0.

How To Add Visual Testing To Existing Selenium Tests

February 27th, 2015 by Dave Haeffner

Thanks again to those of you who attended our recent webinar with Applitools on automated visual testing.  If you want to share it or if you happened to miss it, you can catch the audio and slides hereWe also worked with Selenium expert Dave Haeffner to provide the how-to on the subject. Enjoy his post below.

 

The Problem

In previous write-ups I covered what automated visual testing is and how to do it. Unfortunately, based on the examples demonstrated, it may be unclear how automated visual testing fits into your existing automated testing practice.

Do you need to write and maintain a separate set of tests? What about your existing Selenium tests? What do you do if there isn’t a sufficient library for the programming language you’re currently using?

A Solution

You can rest easy knowing that you can build automated visual testing checks into your existing Selenium tests. By leveraging a third-party platform like Applitools Eyes, this is a simple feat.

And when coupled with Sauce Labs, you can quickly add coverage for those hard to reach browser, device, and platform combinations.

Let’s step through an example.

An Example

NOTE: This example is written in Java with the JUnit testing framework.

Let’s start with an existing Selenium test. A simple one that logs into a website.

// filename: Login.java

import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

public class Login {

    private WebDriver driver;

    @Before
    public void setup() {
        driver =  new FirefoxDriver();
    }

    @Test
    public void succeeded() {
        driver.get("http://the-internet.herokuapp.com/login");
        driver.findElement(By.id("username")).sendKeys("tomsmith");
        driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");
        driver.findElement(By.id("login")).submit();
        Assert.assertTrue("success message should be present after logging in",
                driver.findElement(By.cssSelector(".flash.success")).isDisplayed());
    }

    @After
    public void teardown() {
        driver.quit();
    }
}

In it we’re loading an instance of Firefox, visiting the login page on the-internet, inputting the username & password, submitting the form, asserting that we reached a logged in state, and closing the browser.

Now let’s add in Applitools Eyes support.

If you haven’t already done so, you’ll need to create a free Applitools Eyes account (no credit-card required). You’ll then need to install the Applitools Eyes Java SDK and import it into the test.

// filename: pom.xml

<dependency>
  <groupId>com.applitools</groupId>
  <artifactId>eyes-selenium-java</artifactId>
  <version>RELEASE</version>
</dependency>
// filename: Login.java

import com.applitools.eyes.Eyes;
...

Next, we’ll need to add a variable (to store the instance of Applitools Eyes) and modify our test setup.

// filename: Login.java
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;

    @Before
    public void setup() {
        WebDriver browser =  new FirefoxDriver();
        eyes = new Eyes();
        eyes.setApiKey("YOUR_APPLITOOLS_API_KEY");
        driver = eyes.open(browser, "the-internet", "Login succeeded");
    }
...

Rather than storing the Selenium instance in the driver variable, we’re now storing it in a localbrowser variable and passing it into eyes.open — storing the WebDriver object that eyes.openreturns in the driver variable instead.

This way the Eyes platform will be able to capture what our test is doing when we ask it to capture a screenshot. The Selenium actions in our test will not need to be modified.

Before calling eyes.open we provide the API key (which can be found on your Account Details page in Applitools). When calling eyes.open, we pass it the Selenium instance, the name of the app we’re testing (e.g., "the-internet"), and the name of the test (e.g., "Login succeeded").

Now we’re ready to add some visual checks to our test.

// filename: Login.java
...
    @Test
    public void succeeded() {
        driver.get("http://the-internet.herokuapp.com/login");
        eyes.checkWindow("Login");
        driver.findElement(By.id("username")).sendKeys("tomsmith");
        driver.findElement(By.id("password")).sendKeys("SuperSecretPassword!");
        driver.findElement(By.id("login")).submit();
        eyes.checkWindow("Logged In");
        Assert.assertTrue("success message should be present after logging in",
                driver.findElement(By.cssSelector(".flash.success")).isDisplayed());
        eyes.close();
    }
...

With eyes.checkWindow(); we are specifying when in the test’s workflow we’d like Applitools Eyes to capture a screenshot (along with some description text). For this test we want to check the page before logging in, and then the screen just after logging in — so we use eyes.checkWindow(); two times.

NOTE: These visual checks are effectively doing the same work as the pre-existing assertion (e.g., where we’re asking Selenium if a success notification is displayed and asserting on the Boolean result) — in addition to reviewing other visual aspects of the page. So once we verify that our test is working correctly we can remove this assertion and still be covered.

We end the test with eyes.close. You may feel the urge to place this in teardown, but in addition to closing the session with Eyes, it acts like an assertion. If Eyes finds a failure in the app (or if a baseline image approval is required), then eyes.close will throw an exception; failing the test. So it’s best suited to live in the test.

NOTE: An exceptions from eyes.close will include a URL to the Applitools Eyes job in your test output. The job will include screenshots from each test step and enable you to play back the keystrokes and mouse movements from your Selenium tests.

When an exception gets thrown by eyes.close, the Eyes session will close. But if an exception occurs before eyes.close can fire, the session will remain open. To handle that, we’ll need to add an additional command to our teardown.

// filename: Login.java
...
    @After
    public void teardown() {
        eyes.abortIfNotClosed();
        driver.quit();
    }
}

eyes.abortIfNotClosed(); will make sure the Eyes session terminates properly regardless of what happens in the test.

Now when we run the test, it will execute locally while also performing visual checks in Applitools Eyes.

What About Other Browsers?

If we want to run our test with it’s newly added visual checks against other browsers and operating systems, it’s simple enough to add in Sauce Labs support.

NOTE: If you don’t already have a Sauce Labs account, sign up for a free trial account here.

First we’ll need to import the relevant classes.

// filename: Login.java
...
import org.openqa.selenium.Platform;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
...

We’ll then need to modify the test setup to load a Sauce browser instance (via Selenium Remote) instead of a local Firefox one.

// filename: Login.java
...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", "Login succeeded");
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                "YOUR_SAUCE_USERNAME",
                "YOUR_SAUCE_ACCESS_KEY");
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        eyes = new Eyes();
        eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));
        driver = eyes.open(browser, "the-internet", "Login succeeded");
    }
...

We tell Sauce what we want in our test instance through DesiredCapabilities. The main things we want to specify are the browser, browser version, operating system (OS), and name of the test. You can see a full list of the available browser and OS combinations here.

In order to connect to Sauce, we need to provide an account username and access key. The access key can be found on your account page. These values get concatenated into a URL that points to Sauce’s on-demand Grid.

Once we have the DesiredCapabilities and concatenated URL, we create a Selenium Remote instance with them and store it in a local browser variable. Just like in our previous example, we feedbrowser to eyes.open and store the return object in the driver variable.

Now when we run this test, it will execute against Internet Explorer 8 on Windows XP. You can see the test while it’s running in your Sauce Labs account dashboard. And you can see the images captured on your Applitools account dashboard.

A Small Bit of Cleanup

Both Applitools and Sauce Labs require you to specify a test name. Up until now, we’ve been hard-coding a value. Let’s change it so it gets set automatically.

We can do this by leveraging a JUnit TestWatcher and a public variable.

// filename: Login.java
...
import org.junit.rules.TestRule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;
    public String testName;

    @Rule
    public TestRule watcher = new TestWatcher() {
        protected void starting(Description description) {
            testName = description.getDisplayName();
        }
    };
...

Each time a test starts, the TestWatcher starting function will grab the display name of the test and store it in the testName variable.

Let’s clean up our setup to use this variable instead of a hard-coded value.

// filename: Login.java
...
    @Before
    public void setup() throws Exception {
        DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer();
        capabilities.setCapability("platform", Platform.XP);
        capabilities.setCapability("version", "8");
        capabilities.setCapability("name", testName);
        String sauceUrl = String.format(
                "http://%s:%s@ondemand.saucelabs.com:80/wd/hub",
                System.getenv("SAUCE_USERNAME"),
                System.getenv("SAUCE_ACCESS_KEY"));
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        eyes = new Eyes();
        eyes.setApiKey(System.getenv("APPLITOOLS_API_KEY"));
        driver = eyes.open(browser, "the-internet", testName);
    }
...

Now when we run our test, the name will automatically appear. This will come in handy with additional tests.

One More Thing

When a job fails in Applitools Eyes, it automatically returns a URL for it in the test output. It would be nice if we could also get the Sauce Labs job URL in the output. So let’s add it.

First, we’ll need a public variable to store the session ID of the Selenium job.

// filename: Login.java
...
public class Login {

    private WebDriver driver;
    private Eyes eyes;
    public String testName;
    public String sessionId;
...

Next we’ll add an additional function to TestWatcher that will trigger when there’s a failure. In it, we’ll display the Sauce job URL in standard output.

// filename: Login.java
...
    @Rule
    public TestRule watcher = new TestWatcher() {
        protected void starting(Description description) {
            testName = description.getDisplayName();
        }

        @Override
        protected void failed(Throwable e, Description description) {
            System.out.println(String.format("https://saucelabs.com/tests/%s", sessionId));
        }
    };
...

Lastly, we’ll grab the session ID from the Sauce browser instance just after it’s created.

// filename: Login.java
...
        WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities);
        sessionId = ((RemoteWebDriver) browser).getSessionId().toString();
...

Now when we run our test, if there’s a Selenium failure, a URL to the Sauce job will be returned in the test output.

Expected Outcome

  • Connect to Applitools Eyes
  • Load an instance of Selenium in Sauce Labs
  • Run the test, performing visual checks at specified points
  • Close the Applitools session
  • Close the Sauce Labs session
  • Return a URL to a failed job in either Applitools Eyes or Sauce Labs

Outro

Happy Testing!

 

About Dave Haeffner: Dave is the author of Elemental Selenium (a free, once weekly Selenium tip newsletter that is read by hundreds of testing professionals) as well as a new book, The Selenium Guidebook. He is also the creator and maintainer of ChemistryKit (an open-source Selenium framework). He has helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, and Animoto. He is a founder and co-organizer of the Selenium Hangout and has spoken at numerous conferences and meetups about acceptance testing.

Sauce Labs Appoints Technology Veteran Charles Ramsey as Company’s First Chief Revenue Officer

February 24th, 2015 by Amber Kaplan

SAN FRANCISCO, CA-  Sauce Labs, Inc., the leading cloud-based web and mobile application testing platform, today announced that it has appointed Charles Ramsey as the company’s first Chief Revenue Officer (CRO). Ramsey brings more than 25 years of industry experience and insight to his role at Sauce Labs. He will be responsible for all customer-facing areas, including sales, business development, and marketing, as well as continuing to build on Sauce Labs record 2014 results as the company extends its leadership position in the booming automated testing market.

“Charles is an innovative strategist, committed to building strong relationships with customers and partners,” said Jim Cerna, CEO of Sauce Labs. “His demonstrated ability to strategically grow companies will help us address the exploding demand we are seeing in the market for our technology. With Charles’ experience, leadership, and track record of repeated successes, we are poised to continue our rapid growth trajectory through 2015 and beyond.”

“Sauce Labs is well-positioned to take advantage of the dramatic proliferation of web and mobile apps across a variety of devices and operating systems. I look forward to working with the entire Sauce team to further the company’s market leadership and growth by bringing new levels of innovation, customer experience, and value to the marketplace,” said Ramsey.

Sauce Labs provides an instantly scalable testing cloud that is optimized for continuous integration (CI) and continuous delivery (CD). When tests are automated and run in parallel on virtual machines across multiple browsers and platforms, testing time is reduced and developer time is freed up from managing infrastructure. When paired with a CI system, developers can easily test web, hybrid and native applications early on in their development cycles, continuously and affordably. Sauce Labs currently supports more than 480 browser, operating system and device platform combinations.

Prior to joining Sauce Labs, Ramsey was an early member of the Quest Software management team, where he served as vice president of World Wide Marketing and Sales. He is a former Venture Partner at JMI Equity and has also served on the Board of Directors at notable companies such as Configuresoft, Inc. and ServiceNow, Inc. Early in his career, Ramsey rose to vice president, North America Sales for Computer Intelligence, a division of Ziff Davis, after beginning his career with the IBM National Accounts Division in a variety of sales assignments. He has a Bachelor of Arts from the University of California, San Diego and a Master of Information Management from the American Graduate School of International Management.

Helpful Links

About Sauce Labs
Sauce Labs is the leading cloud-based web and mobile application automated testing platform. Its secure and reliable testing infrastructure enables users to run JavaScript unit and functional tests written with Selenium and Appium, eliminating the time and expense of maintaining a test grid. With Sauce Labs, organizations can achieve success with continuous integration and delivery, increase developer productivity and reduce infrastructure costs for software teams of all sizes.

Sauce Labs is a privately-held company funded by Toba Capital, Salesforce Ventures, Triage Ventures and the Contrarian Group. For more information, please visit http://saucelabs.com.