Posts Tagged ‘CI’

The Benefits of Parallel Testing

August 6th, 2015 by Greg Sypolt

Running in slow motion?

Are you running, but can’t make your feet move as fast as you want them to? This is a common feeling among beginners, as well as experienced automation developers. As your regression suite grows, it takes longer to run tests, and soon you have a problem because your regression suite is running longer and longer. There are a few approaches to smarter testing: reduce the number of tests, only run the tests applicable to the change, or optimize the test execution.

Marching toward continuous integration

As your software development team marches toward Continuous Integration (CI), the process involves a lot of automated testing. Even automated testing consumes precious time, so you need to ensure your automated tests are designed to scale. If a particular change is going to cause one or more tests to fail, the team needs to know about it as quickly as possible. Developing scripts to be lean and independent allows for fast feedback to developers.

Let’s unleash the power of parallelization

Use parallelization to speed up slow automated UI tests, and set a standard to develop lean and independent starts. A single cucumber scenario can easily take minutes to run. When you have a lot of scenarios, they can quickly compound your suite and take several minutes or hours to complete. No one wants slow automated tests — tests so slow that they only run a couple of times per day. Everyone expects automated tests to be launched for every build and send feedback within minutes, not hours. Set a standard for every build: the test execution must complete and send feedback within 10 minutes. (more…)

Test Automation KPIs

June 3rd, 2015 by Greg Sypolt

One of the interesting things about automation is that it frees you up from time-intensive manual testing, allowing you to spend time on strategic elements—because if you do not spend time on strategy, your capabilities as a team will not grow. And part of that growth means focusing on valuable metrics— metrics that will help you learn, and improve your processes.

Once you have processes in place, the next crucial step is to invest in automation. Automation helps you work faster, and makes your work consistent, traceable, and shareable, which is also imperative. All this comes only after establishing the right KPIs (key performance indicators).

Automation: Deliver Faster, from Months to Minutes

Ask yourself this question: without CI (continuous integration), how long would it take your organization to deploy a change that involves just one line of code? For instance, say your organization sets an objective to deploy a change in production within 30 minutes. To achieve this objective, everyone has to agree on the tools and processes that are needed for an easy button approach (aka continuous integration).

Let’s review the roles, team responsibilities, and the CI process. (more…)

Best Practices in Mobile CI [WEBINAR]

April 22nd, 2015 by Bill McGee

Modern organizations today feel immense pressure to deliver better software faster, and this is no different in the mobile space. The best practice of Continuous Integration for web dev has been embraced for years as it is a proven mechanism that accelerates production cycles. However,mobile developers have been slow to adopt CI, despite needing a quick go-to-market plan.

In large part, this is because mobile brings with it a set of unique challenges that make implementation tough. Nevertheless, tools have evolved and mobile development teams now have many options to choose from to implement a solid mobile CI system.

In our next webinar, Kevin Rohling (Emberlight, and Kristian Meier (Sauce Labs) will cover best practices in implementing a mobile CI system and demonstrate how you can easily build, test, and deploy mobile apps.

This webinar will cover: (more…)

Guest Post: All About Mobile Continuous Integration

April 20th, 2015 by Bill McGee

Image credit: Loris Grillet

CI’s not a new thing. Wikipedia says the phrase was first used back in 1994, way before modern mobile apps. Today it’s commonplace in many dev shops for developers to expect that their code is automatically tested when they commit and even automatically deployed to a staging environment.

For mobile developers though, it’s been a slow road to adopting many of these same practices.  In large part, this is because mobile brings with it a whole set of unique challenges that make implementation tough.  Nevertheless, tools have evolved a lot and mobile dev teams get a lot of value and goodness from having a solid CI system in place. Here are my top 3 reasons for using CI with the mobile products I work on: (more…)

Speed up Selenium with Application Mocks

April 8th, 2015 by Chris Riley

Let’s face it, we all know Selenium is slow. Not the Selenium scripts themselves, but the process of interacting with the operating system and browser drivers, in addition to your application’s own responsiveness. Luckily for us, there’s a great way to speed up functional testing in Selenium by combining a comprehensive parallel testing grid with application mocks. (more…)

Repost: Testing in a Real Browser with Sauce Labs + Travis CI

March 26th, 2015 by Bill McGee

This post comes from our friend Sam Saccone, who wrote a nice how-to on using Sauce Connect and Travis CI. Check out the original post on his blog.

I recently found myself implementing a basic set of JavaScript integration tests on an open source project. I quickly came to the realization that there is a serious lack of good documentation on how to get a basic browser test running on a Travis CI using Sauce Labs and Sauce Connect.

After stumbling across the barren desert of outdated doc pages, incorrect stackoverflow answers, and ancient google group posting, I present to you the spoils of my quest to the underbelly of the web.

Let’s approach this in the context of a real problem: Testing javascript in a real web browser. (more…)

Repost: Lessons Learned from Automating iOS Apps – What To Do When Tests Require Camera Roll Resources?

March 18th, 2015 by Bill McGee

This post comes from our friend Jay Sirju at Animoto, who is leveraging Sauce, Appium, and CI methodologies to automate mobile testing for their iOS application.  Check out the original post on their blog.

Here at Animoto, the mobile application development team had spent some time over the past year investigating and implementing CI methodologies into the development cycle of the Animoto Video Maker application for iOS. A major part of this initiative involved creating automated test cases that would run at various times and circumstances.

First a bit of background. When we started this, we had already implemented a good amount of automation for the Animoto website. We had chosen to use Selenium and ran our automated tests against various browsers using Sauce Labs. We decided to extend our existing infrastructure to support running automated tests using theAppium library against Sauce Labs.   For those unfamiliar with mobile testing on Sauce Labs, they use the iOS and Android Simulators to run tests. I know, not ideal, but we can get to that another time.

The Problem

For anyone who has ever launched a fresh iOS Simulator (before Xcode 6), the OS is in it’s factory state. The Animoto Video Maker App transforms your pictures, video, and text, into professional looking videos… see where I’m going?


A lot of user flows depend on having some photos in the camera roll. A factory-fresh simulator without any photos means there are man flows we can’t automate. Unfortunately, at the time of writing, Sauce Labs does not have a way to upload assets to populate the Camera Roll, and the simulators are reset after executing each test case. Starting with XCode 6, the Camera Roll does have some images out of the box, but what was really needed were meaningful pictures and videos. So what is needed is a way to populate the Camera Roll while working within the constraints of running tests in Sauce Labs. Well, the app already reads images and pictures from Camera Roll, what about writing to it as well?

Adding or Altering Configuration Profile

Before we get to actually populating the Camera Roll, we need a mechanism to ensure that this logic is performed only when the intent is to run automation. Out of the box, Xcode provides 3 build configurations (Debug, Release, and Distribution). We can edit these configurations, add new ones, or even delete unnecessary ones. In this case, we can simply add a Test configuration to the mix. Once we did that, we were able to change various build settings to help create hooks for app automation. We can start out by adding a Preprocessor Macro for the Test build configuration, so that we can tell the pre-processor when to compile test hooks into the build.

animoto 2

Okay, now we can do some fun stuff with the build. For the sake of brevity, let’s focus specifically on the original issue: Getting pictures and videos into the Camera Roll.

Change Test Configuration Profile Settings

First things first – how does one get pictures and videos up to Sauce Labs? We can simply add them to the iOS project, but that would increase the size of the application bundle regardless of which build configuration is being used. Definitely not ideal. A better choice would be to store them somewhere externally and copy them to the application bundle when the Test configuration is used. This can be done by running a script when building the project.

[code language=”ruby”]if [ ${CONFIGURATION} == "Test" ]; then

Populating the Camera Roll

Now we have an application bundle that contains a bunch of sample pictures and videos. This is great because when the application gets uploaded to Sauce Labs for testing, so do all the sample data.   The following code example assumes all the sample images are in a folder within the application bundle named ‘TestImages’:

[code language=”ruby”]+ (void) populateCameraRoll
NSString* absolutePicturePath = [[NSBundle mainBundle] pathForResource:@"TestImages" ofType:nil];
NSArray* pictureList = [[NSFileManager defaultManager] contentsOfDirectoryAtPath:absolutePicturePath error:nil];

for(int i = 0; i < [pictureList count]; i++)
NSString* absolutePictureFilePath =[ NSString stringWithFormat:@"/%@/%@", absolutePicturePath,[pictureList objectAtIndex: i]];

NSData *jpeg = [NSData dataWithContentsOfFile:absolutePictureFilePath];

UIImage *image = [UIImage imageNamed:absolutePictureFilePath];

CGImageSourceRef source = CGImageSourceCreateWithData((__bridge CFDataRef)jpeg, NULL);
CFDictionaryRef imageMetaDataRef = CGImageSourceCopyPropertiesAtIndex(source,0,NULL);
NSDictionary *imageMetadata = CFBridgingRelease(imageMetaDataRef);

if (image != nil)
ALAssetsLibrary* library = [[ALAssetsLibrary alloc] init];

dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
dispatch_semaphore_t sema = dispatch_semaphore_create(0);

[library writeImageToSavedPhotosAlbum:[image CGImage] metadata:imageMetadata completionBlock:^(NSURL *assetURL, NSError *error)
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);

The above code makes a bunch of assumptions. First, it only works for images. Another folder consisting of sample videos can be created, and the using similar logic and the writeImageToSavedPhotosAlbum method. Next, calling writeImageToSavedPhotosAlbum can indeed fail. For the sake of keeping this example code more readable, error handling was excluded. Retry logic should be included if an error is returned.

Finally, you may have noticed the use of a semaphore in the example code. Writing images to the Camera Roll is actually an asynchronous call, meaning that the call returns immediately while a separate thread processes writing the image data to the Camera Roll. The writeImageToSavedPhotosAlbum method can fail if there are too many threads trying to write image data simultaneously. The semaphore is used ensure that images are written to the Camera Roll sequentially. This makes using the writeImageToSavedPhotosAlbum method much more stable.

Okay, so now that is left is to call the method when running the Test configuration. This can easily be done using the Preprocessor Macro setting that was above mentioned.

[code language=”ruby”]#if TEST
[ANTestClass populateCameraRoll];

It is recommended to call the method somewhere deterministic (ie -a button tap). Simply populating the Camera Roll at start-up may mess up launching Apples Instrumentation library because of the Alert displayed when the app accesses the Camera Roll for the first time. This lesson was learned the hard way.

This opens up the ability to add more automated test hooks into the application under test, but as a word of warning; the more hooks added, the more the test configuration of the app diverges from what is being released to customers.


Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Re-Blog: CI & CD With Docker, Beanstalk, CircleCI, Slack, & Gantree

December 10th, 2014 by Bill McGee


This is a follow-up post to a series highlighting Bleacher Report’s continuous integration and delivery methodology by Felix Rodriguez. To find out how they were previously handling their stack, visit the first, second, and third posts from June 2014. 

There is definitely a huge Docker movement going on in the dev world right now and not many QA Engineers have gotten their hands dirty with the technology yet. What makes Docker so awesome is the ability to ship a container and almost guarantee its functionality. (more…)

Announcing CircleCI Integration on Sauce Labs

August 11th, 2014 by Bill McGee

Last week our friends at CircleCI showed us how to securely run cross-browser front-end tests on the Sauce Labs cloud using their hosted Continuous Integration service. We’ve long been advocates of good continuous integration practices and have developed a few plugins for some of the more common CI servers out there. We’re super excited to add CircleCI to our list and even more excited about how easy it is to get it going!

Continuous Integration in the Cloud

Continuous Integration, if you don’t already know, is the process of building your app frequently from a shared source code repository and running tests to make sure everything works. If your tests don’t pass and the build is not successful, the code that was checked in since the last good build is where the defects were introduced, and so problems are much easier to find and fix quickly.

Maintaining a local CI server can be a hassle. Anyone who’s spent any considerable time configuring Jenkins jobs with all it’s various plugins and tasks can tell you all about it. CircleCI, on the other hand, integrates directly with GitHub and can actually *infer* necessary settings directly from your code (if you’re following good development practices for that language and environment) and so many projects just magically build themselves on CircleCI without any additional configuration on your part. It’s like three clicks from zero to CI. Pretty sweet! If you do need to tweak or customize any settings, you can easily do so by describing those settings in a circle.yml file placed in your repo.

Running Tests on Sauce Labs Browsers

Sauce Labs is the world’s largest cross-browser client cloud. We host over 375 different configurations of operating systems and browsers so you can ensure that your app works on all the specific platforms and versions you need to support. These days that’s an ever-growing list! So it makes sense to run these tests with your continuous integration process so you know things work across the board and you don’t end up spending a bunch of time and trouble trying to hunt down bugs that were introduced much earlier in the development cycle.

Now, if your build deploys your code to a publicly accessible environment, CircleCI will simply execute your Selenium tests and you probably won’t need to configure anything, since Sauce Labs browsers will be able connect to that environment over the public network. However, if you want CircleCI to execute your tests locally in it’s build containers, you’ll need to use Sauce Connect.

Sauce Connect is a secure tunneling utility which allows you to execute tests behind firewalls via an encrypted connection between Sauce Labs and your testing environment. When you run Sauce Connect yourself, you typically do it from a command line or shell script and supply your Sauce Labs account credentials as parameters. With a CircleCI build, you’ll need to set it up in the circle.yml file so it can be launched as part of the build process and those tests can run inside the local container.

All that’s really involved here is telling the build task where to find Sauce Connect and how to start it up with your account credentials.

The first part is downloading and unpacking the Sauce Connect file, which you add as a custom dependency entry in your circle.yml:

		   - wget
		   - tar -xzf sc-latest-linux.tar.gz

The second part is to add your credentials, launch the tunnel, and check that it’s running before kicking off the tests. You’ll put these lines in the `test` section of circle.yml:

        - ./bin/sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY -f ~/sc_ready:
            background: true
            pwd: sc-*-linux
        # Wait for tunnel to be ready
        - while [ ! -e ~/sc_ready ]; do sleep 1; done

That’s all there is to it. You can find out the details here and see a full example on GitHub. And CircleCI has a nice little utility to help you add your credentials as environment variables so that they are not visible as plain text in the repo.

With CircleCI tackling all the necessary work involved in supporting your continuous integration process and Sauce Labs hosting the world’s most extensive cross-browser client cloud, you can be free of the costs and hassles of managing all these systems and grids and get back to focusing on the business of making great software.

Michael Sage, Principal Technology Evanglist, Sauce Labs

Michael Sage is a Principal Technology Evangelist at Sauce Labs. He helps software teams develop, deliver, and care for great apps. He’s lived through two or three technology revolutions and expects a few more to come, and finds the prospect incredibly exciting. A native of Philadelphia, he’s made San Francisco his home for over 25 years, but still can’t find a good hoagie there.

Bleacher Report’s Continuous Integration & Delivery Methodology: Creating an Integration Testing Server

June 10th, 2014 by Bill McGee

Bleacher-report-logoThis is the second of a three part series highlighting Bleacher Report’s continuous integration and delivery methodology by Felix Rodriguez.  Read the first post here.

Last week we discussed how to continuously deliver the latest version of your application to a staging server using Elastic Beanstalk. This week we will be discussing how Bleacher Report continuously runs integration tests immediately after the new version of our app has been deployed.

When our deploy is complete, we use a gem called Slackr to post a message in our #deploys chat room. This is simple enough and just about any chat software can do this. We chose to use Slack because of the built-in integration functionality.

We created an outgoing webhook that submits any posts to our #deploys channel as a post to our Cukebot server. The Chukebot server searches the text, checks for a “completed a deploy” message, then parses the message as a Json object that includes the deploy_id, user, repo, environment, branch, and Github hash.

class Parser
  ## Sample Input:
  # OGUXYCDI: Dan has completed a deploy of nikse/master-15551-the-web-frontpage-redux to stag_br5. Github Hash is 96dd307. Took 5 mins and 25 secs
  def self.slack(params)
    text = (params["text"])
    params["deploy_id"] = text.match(/^(.*):/)[1]
    params["branch"] = text.match(/of\s(.*)\sto/)[1]
    params["repo"] = text.match(/to.*_(.*?)\d\./)[1]
    params["cluster"] = text.match(/to(.*?)_.*\d\./)[1]
    params["env"] = text.match(/to\s.*_.*?(\d)\./)[1]
    params["suite"] = set_suite(params["repo"]) 
    params["hash"] = text.match(/is\s(.*?)\./)[1]
    puts params.inspect
    return params

Once parsed, we have all the information we need to submit and initiate a test suite run. A test suite and its contained tests are then recorded into our postgresql database.

Here is an example of what this suite would look like:

  id: 113,
  suite: "sanity",
  deploy_id: "FJBETJTY",
  status: "running",
  branch: "master",
  repo: "br",
  env: "4",
  all_passed: null,
  cluster: " stag",
  failure_log: null,
  last_hash: "0de4790"

Each test for that suite is stored in relation to the suite like so:

  id: 1151,
  name: "Live Blog - Has no 500s",
  url: "",
  session_id: "20b9a64d66ad4f00b21bcab574783d73",
  passed: true,
  suite_id: 113
  id: 1152,
  name: "Writer HQ - All Article Types Shown",
  url: "",
  session_id: "4edbe941fdd8461ab6d6332ab8618208",
  passed: true,
  suite_id: 113

This allows us to keep a record over time of every single test that was run and to which suite and deploy it belongs. We can get as granular as the exact code change using the Github hash and later include screenshots of the run. We also have a couple of different endpoints we can check for failed tests in a suite only, tests that have passed only, or the last test suite to run on an environment. We wanted to record everything in order to analyze our test data and create even more integrations.

This helps us automatically listen for those completed deploy messages we talked about earlier, as well as to have a way of tracking those tests runs later. After every test suite run we then post the permalink of the suite back into our #cukes chat room so that we have visibility across the company.

Another added benefit is that it allowed us to build a front end for non tech savvy people to initiate a test suite run on any environment.

Check it out for yourself; we just open sourced it.

Stay tuned next week for part two of this mini series! You can follow Felix on Twitter at .

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.