Dealing With Test Log Data

October 26th, 2016 by Michael Churchman

Test logs. What are they good for? What can you do with them? What should you do with them? These aren’t always easy questions to answer, but in this post, we’ll take a look at what’s possible and what’s advisable when it comes to testing log data.

What are Test Logs Good For?

What are test logs good for? Or are they good for anything at all?

Let’s start with an even more basic question: What is (and what isn’t) a testing log? A testing log is not simply test output. Minimal pass/fail output may log the results of testing, but a true testing log should do more than that. At the very least, it should log the basics of the testing process itself, including test files used, specific test steps as they are performed, and any output messages or flags, with timestamps for each of these items. Ideally, it should also log key processes and variables indicating the state of the system before, during, and after the test. (more…)

Avoiding the UI: Why and How to Run Tests With Scripts

October 4th, 2016 by Michael Churchman

There’s no doubt about it: a user interface (whether it’s graphic or text-only) can be very nice, at least when you need to make decisions in real time or enter data on the spot.

But when you know exactly what you’re going to do and how you’re going to perform each step, and you have a set of tasks that you’re likely to perform more than once or twice, any kind of user interface can slow you down, get in the way, and eventually become a maddening, time-wasting annoyance. This is nowhere more true than with testing, where complex and highly repetitive tasks are the norm, and there is nothing to be gained by waiting to enter an occasional command or piece of data.

This article explains the value of test scripts, and the circumstances under which you should consider using them to perform software tests. (more…)

Software Testing Tools for Your QA Team

September 27th, 2016 by Chris Riley

Ashley Hunsberger, Greg Sypolt and Chris Riley contributed to this post.

Software testing tools are a vital resource for every successful QA team. But with so many tools and testing frameworks out there – from Selenium and Protractor to Espresso and Xcode – how do you choose which are best? How should your toolset vary depending on whether you do desktop testing, mobile testing, or both? And how do you make the most of software testing tools?

Below are answers to these questions from the panelists of a recent Sauce Labs webinar focused on software testing and QA. The webinar was hosted by Chris Riley, with Ashley Hunsberger and Greg Sypolt serving as panelists. You can also find their recommendations on software testing tools below.

Which tools has Greg used for test automation at Gannett?

Greg: Here’s an inventory of testing frameworks and tooling used across Gannett products (technology alignment): (more…)

Team Building and Quality Assurance

September 22nd, 2016 by Chris Riley

Ashley Hunsberger, Greg Sypolt and Chris Riley contributed to this post.

How do you build an effective team of quality assurance (QA) engineers? Where do you look to recruit the best QA professionals? How should you integrate your QA team within the rest of your organization? These and other questions related to the topic of team building came up during a recent webinar hosted by Chris Riley, Ashley Hunsberger, and Greg Sypolt. Here are links to the first and second post in this series.

Team building is an essential part of an effective QA operation, of course. Your testing strategy and execution are only as strong as the people who design and conduct them. And in today’s IT world, where DevOps is changing the way different types of engineers interact, and the supply of good QA professionals outstrips demand, having a well-planned and executed approach to team building is crucial.

Below are some of the most important team-building questions that came up during the webinar, along with answers from participants.

One thing I have noticed lately is that QA engineers are getting more respect. With automation, their skillset looks more like those of a developer. Do you agree? (more…)

Making Your App Testable

September 16th, 2016 by Dan Cuellar

When writing test automation, one of the most important factors for determining the amount of time and resources you will consume (and ultimately the success or failure of the endeavor) is the testability of your application. By testability, I’m referring to how the app interacts with UI (and other) automation frameworks, the ease by which a test script can setup the scenarios you wish to test, and how you make your tests safe for concurrency.

Making Elements Accessible

Let’s address the matter of controlling your application with automation frameworks first. Given that you are reading this blog, it is likely you are using either Selenium or Appium as your automation framework, so this blog post will only address these frameworks.

Web Content (Selenium)

Let’s start with Selenium. In order for your web app to be easily controlled by Selenium, you need to think ahead as to how you will identify important pieces of the DOM when constructing tests. Selenium provides many means by which you can do this, called “Locator Strategies”. Some are better than others. You should consider which of these you will be using in your tests as you develop the user interface. Ideally, each element would have an ID attribute applied directly to the tag for any element that the test will exercise.

Sometimes, for one reason or another, you may not be able to use an ID. If this is the case, the next recommended technique is to use a CSS selector. If your web app is developed with good principles, such as BEM (Block Element Modifier), it is likely easier to automate as it should have a relatively short globally unique CSS selector. If it does not, I would not recommend adding a CSS class just for automation, as it isn’t the purpose of the styling language to do automation. Rather, I would suggest that you use an HTML  data-* attribute. You can use the CSS selector locator to grab an element by one of these attributes, but the good news is you aren’t adding unnecessary classes to your CSS, and the purpose of the HTML attribute will be much clearer. (more…)

How Often Should You Parallel Test?

September 9th, 2016 by Michael Churchman

How often should you parallel test? If that sounds like a trick question, maybe it is. In this post, we’ll let you in on the “trick” part of the question, and then we’ll talk about what really matters when it comes to when and how often you should parallel test.

Parallel Testing – What is it?

First, the trick. It lies in what parallel testing is, and more to the point, what it isn’t.

What is parallel testing? The term “parallel testing” is generic and rather broad, but it typically refers to automated systems for simultaneously testing multiple applications or components, with each application or subcomponent being tested on a different computer. It is sometimes also used to describe automated testing of a single application or component on multiple platforms. The test computers can be individual hardware units, or more typically, separate virtual machines. In all cases, however, the combination of automation and multiple test systems makes it possible to run many more tests than would be practical with serial testing, and it reduces the time required for testing to a fraction of that required for the equivalent serial tests. The key points to keep in mind about parallel testing are that: (more…)

A Getting Started Guide to Setting Up Jenkins

September 7th, 2016 by Greg Sypolt

The goal of this getting started guide is to help teams get Jenkins continuous integration (CI) servers configured, and discover how to make a newly deployed CI infrastructure fully operational. Jenkins is a leading open source CI server. It is flexible, providing hundreds of plugins to support building, testing, and deployment, and is capable of automating any project. Jenkins CI infrastructure can be deployed to on-prem, in the cloud using configuration management tools, and third-party vendor. For the purpose of this article, let’s assume that our Jenkins CI servers are deployed in the cloud and focus on configuration of Jenkins’ web interface. I will walk through various processes and steps to set up a new Jenkins CI server for production.

Recommended best practices for CI architecture

Don’t jump head first into configuration and creating a pipeline without planning, designing and establishing standards for your CI architecture. Taking the time to think about infrastructure first will enable a stable and restorable infrastructure. Let’s review a few recommended best practices to consider in your future CI pipeline.

Backup CI Server (Fail Safe)

It may seem obvious. Recommend setting up a backup process for Jenkins configuration. Script a Jenkins job to use the thinBackup plugin or S3 plugin to send the Jenkins configuration to an Amazon S3 (cloud storage).

Pipeline Configuration

Here are recommendations to consider: set environment variables (i.e., hidden passwords, SSH keys, API keys, etc.); security—create generic reusable jobs, naming conventions (i.e., jobs and environment variables); keep jobs small—modulization, scalable infrastructure that allows for auto-scaling of Slave nodes. (more…)

The Cost of a Reject

September 1st, 2016 by Joe Nolan

You’ve heard about the cost of a bug—The longer it takes for a bug to be discovered, the more expensive it is to fix. But have you ever considered the cost of a reject? Do you have visibility into how much rejected stories and bugs are affecting your Scrum team’s capacity in terms of time?

Let’s take a look at just how impactful rejects can be, and how to mitigate them when they occur.

The Cost

Start with an average story or bug ticket in a hypothetical Scrum team:

  • A story or bug ticket is created and assigned to the sprint.
    • The bug has steps to reproduce—if the writer was efficient.
    • For the story, QA and the Scrum team determine the test strategy.
  • The developer works on the ticket and pushes it to QA.
  • QA verifies or rejects the ticket.

Different tickets take different levels of effort. For the sake of this article, let’s assume it takes an average of 30 mins for QA to verify the ticket.

At least, that’s what happens when all goes well. (more…)

How to Avoid Thread.Sleep in Test Automation

August 30th, 2016 by Sahas Subramanian

Several years back, I wanted to understand if it’s really necessary to use thread.sleep in test automation code. Here is what stuck in my mind: Thread.Sleep(n) means blocking the current thread for the number of time slices that occur within “n” milliseconds.

There are a few things to consider before using thread.sleep in your test automation code. They include:

  1. The thread time slice varies from OS to OS and processor to processor. This means that once your thread goes to sleep, it’s going to wait for its turn to come back into the scheduler. So it’s highly likely that the thread.sleep time means more than what you really intended. For example, with thread.sleep(1000), you intended 1,000 milliseconds, but it could potentially sleep for more than 1,000 milliseconds too as it waits for its turn in the scheduler.
  2. Each thread has its own use of CPU and virtual memory. That means our thread acquired its resources (say 1MB virtual memory, some CPU cycles and some context switching cycles), and is simply not using them.
  3. If it’s a foreground thread, it’s preventing the application from exiting as well.
  4. What about Thread.Sleep(0)? This tells the processor that I’m ready to lose the remainder of my time slice, and you may pick up the next thread from the queue. On one hand, it feels good and efficient. But on the other hand, you are also telling the processor that you are better at scheduling than the processor. (I don’t know the side effects this can cause, but I’m not smarter than the processor.)


We are talking about milliseconds when dealing with thread.sleep, but it’s important to understand that the more we use thread.sleep, the more we defer solving the real problem, which is how to handle async in modern web apps. (more…)

The Sauce Journey – Shu Ha Ri

August 25th, 2016 by Joe Alfaro

If you’re attempting to implement an Agile/Scrum development process where none has existed before, you will surely an encounter a moment of frustration on the part of your developers. “Why do we have to do these standups?” “I don’t understand why we need to assign story points, can’t we just get to the projects?” “Where is my technical specification?” Like Ralph Macchio in The Karate Kid, your developers may wonder why you have them doing the engineering equivalent of “wax on, wax off,” when what they really want to do is get into the fight. What Ralph Macchio eventually understands is that the performance of rote, rigid external exercises is a first step on the road to internal mastery, a process well known in the world of martial arts as Shu Ha Ri.

In its broader definitions, Shu Ha Ri describes a process of learning: in the Shu stage, the learner follows directions literally and adheres rigidly to whatever rules the teacher has set. In the Ha stage, the learner begins to see how the rules and directions can be adapted for specific situations, and exercises some judgement in how they should be applied. In the Ri stage, the learner has developed her own techniques, and now innovates freely as the situation demands. (more…)