Why is Manual QA Still So Prevalent?

April 28th, 2016 by Joe Nolan

This past week I casually heard comments alluding to the imminent death of the QA Analyst or Manual Tester. (To be clear, I am not referring to the QA Automation Engineer, who builds test automation.)

Not only does the function not seem to be going away, recruiters are still out their hunting testers down. Out of curiosity I did my own review of randomly selected job posts from Monster and Indeed for average QA positions, and discovered that there are still a lot of jobs available for manual testers.

With the importance of catching bugs early, and the ability to automate all testing, why do companies and projects resist the investment in CI and test automation? I will explore the reasons why now, and whether the resistance is good or bad. (more…)

Mobile vs. Web: Which is Harder to Test?

April 19th, 2016 by Joe Nolan

Have you ever worked on a web-based test team and switched to a mobile team and wondered if your life is about to get easier or harder? There are significant differences between testing mobile vs. web, and yes, one is MUCH harder than the other. Want to guess which one? Read on and see if you guessed correctly.

Let’s Compare

The table below shows the different facets of testing and where its execution is most challenging. (more…)

Test Faster and Smarter by Testing in Production

April 14th, 2016 by Chris Riley

You may dread the term testing in production (TiP). The thought of potential loss of data, downtime, and a damaged reputation to organizations can be daunting. But things need not be that way. In fact, today, testing in production is used by some of the biggest organizations with much success. But can it become a reality for your team?

Accident or Intentional?

Testing in production is not a completely new concept. In fact, you’ve probably seen it in action more often than you imagine. Think of an app that you’ve released or one that you know of that was poorly tested. You likely spent the next few weeks firefighting, and got it to be functional faster than you thought possible. In this case, you were forced to test in production. But, what if you could hone the art of testing in production and use it to your advantage? What if you could spot and fix issues so much faster than you do today? What if you could influence development from start to end? What if you could do all this without risking the reputation of your team or organization? That’s the promise of TiP, and it’s worth a second glance. (more…)

Waiting for Green

April 12th, 2016 by Ashley Hunsberger

Every now and then, you may encounter a time when you need to stabilize your automated UI tests (for myself, that time is now). Although you don’t want to add to a framework that you are stabilizing, you probably don’t want to halt development on new features. (Warning — telling your leadership team no one is allowed to add more tests until everything goes green might not go over well.)

What do you do in the meantime? The answer is simple, and I look to some practices in Behavior Driven Development (BDD) as a guide – build a test skeleton into your current framework.

The First Rule of Stabilization: Don’t Create a Manual Test Suite

While you may temporarily need to revert to manual execution, it does not necessarily mean you’ll want to go back to a manual test suite for a couple of reasons: (more…)

Get Started with Protractor Testing for AngularJS

April 6th, 2016 by Greg Sypolt

How do you test your AngularJS applications? With Protractor. Protractor is an end-to-end testing framework for AngularJS applications. This getting started guide is for new software developers in test who are interested in learning about Protractor testing. By following this getting started guide, you’ll understand how to build a firm foundation and learn fundamental Protractor testing for AngularJS applications.

Build a solid foundation

To build better software tests, you’ll need a solid base of the technologies behind your application, which teaches you the principles that will be essential when coding, executing and debugging your tests. Let’s focus on the correct learning path for Protractor testing. This is the age of JavaScript (JS) applications and frameworks. JavaScript is the foundation and critical learning block needed to be successful with Protractor testing, and what follows is your roadmap to learn JavaScript, NodeJS, and AngularJS for Protractor testing. (more…)

Implied Testing

March 30th, 2016 by Joe Nolan

Implied Testing is a way to write a test that indicates other parts of your workflow are working as you try to accomplish a goal. Make use of Implied Testing to minimize the amount of documentation and testing artifacts on a project.

According to the Manifesto for Agile Software Development, we should favor working software over comprehensive documentation. While this sounds good in theory, all too often test teams are asked to produce documentation explaining what they plan to test (in detail). The concept of Implied Testing will help save a lot of writing as it will eliminate duplication, and streamline the tests that feed into automation, allowing for simpler, more re-usable scripts.

Why Do We Write Detailed Tests?

In an ideal Agile world we limit our test documentation and focus on automated tests, with manual smoke and exploratory tests to supplement them. Our Acceptance Criteria in the stories should guide the tests necessary for the story to be complete. Unfortunately, circumstances can require an extensive amount of test documentation artifacts to be produced. Why? (more…)

Fragile Automation

March 28th, 2016 by Ashley Hunsberger

User Interface (UI) Testing.

The idea is simple — automate some UI tests to ensure your application is still behaving as expected. Usually your first set of tests — running green, no doubt — make you all cheer and pat yourselves on the back. Then you open up the framework to more people. Despite the reviews (so many reviews), the failures start to come, and they don’t stop. Or they run green and then fail and then run green again. And then fail again. So why are they so unstable? Is it bad scripts? Environment issues? Sometimes you just don’t know, and you think you are going to lose your mind. Let’s take a look at some common and potential issues you may be facing.

Architecture, Environments and Settings

Is your infrastructure designed for stability? Are you using on-premise or cloud instances? What may have saved you a dollar upfront could cost you many more down the road – so your testing environment is important.

Understand if your tests require particular system settings. Tests failing because of unwanted server variables is a waste of everyone’s time. We found that out the hard way a long time ago. You may need to have some isolated tests that cannot be run on the same server so the majority of your other tests can pass. (Or maybe decide how important the test really is.)

Or let’s say your testing frameworks are stable, but what about the tools or libraries you are importing? Are you pinning your stack to a version of these tools or libraries? A new version can completely break everything. (more…)

3 Simple Strategies to Get Started With Automation

March 24th, 2016 by Joe Nolan

If your test automation team’s directive is to automate X amount of tests, and you have no strategy as to which tests they should focus on, you are wasting your time. Before you begin writing your first line of automation code, make sure you have a strategy in place. Otherwise, you will have a ton of ineffective tests to maintain.

Don’t Choose a Random Goal

How many times have you been told that the goal of the team is to have X amount of test coverage? This is an arbitrary value picked out of the sky. What is it based on? If a UI automation team were to cover 80% of the stories in a sprint, they would never get done in time.

We all know how fragile UI automation is! How many times will a designer make a change that directly affects the UI and breaks the test? This is almost manageable during a sprint while you are working closely together, but how about when the product is sent to be translated to another language? The translator inevitably comes back with suggestions to allow for phrases more common and translatable. Bugs might be entered and UI changes made by a maintenance team with no heads up to the automation team, and Bam! — You have broken tests that need to be investigated. (more…)

A Two-Minute BDD Overview

March 16th, 2016 by Ashley Hunsberger

Behavior Driven Development, or BDD, can help get your teams building the RIGHT product. Although I’ve heard the term used interchangeably with Test Driven Development (TDD), I personally see it as an extension of TDD to help your team focus on the business’ goals. While TDD provides tests that drive development, those tests may or may not be helping you meet those goals.

The WHY Behind the Code: BDD vs. TDD

Behavior Driven Development
Test Driven Development
  • Start with business value, then drill down to feature sets
  • Lots of tests that may or may not meet the business value
  • Team gets feedback from the Product Owner
  • Coder gets feedback from the code

BDD is a more outside-in approach that is really focused on business drivers. It takes TDD a step further (as you still want that feedback from the code), but it now gives you feedback on the feature.


BDD Workflow

Getting Started

So what is the general process to use BDD? It aligns itself nicely in the Agile framework and is simply a WAY to implement Agile. Continue with your usual scrum activities: milestone planning, defining user stories and acceptance criteria and developing, and iteratively repeat that process until you are ready to release. (more…)

Write Great Cucumber Tests

March 14th, 2016 by Greg Sypolt

Using Cucumber with outlined best practices in your automated tests ensures that your automation experience will be successful and that you’ll get the maximum return on investment (ROI). Let’s review some important best practices needed before you start developing Cucumber tests.

Writing Features

Feature files help non-technical stakeholders engage and understand testing, resulting in collaboration and bridging the communication gap. For this reason, well-written feature files can be a valuable tool for ensuring teams use the BDD process effectively.

Here are some suggested standards for Cucumber Feature files:

OrganizationFeature files can live at the root of the /features directory. However, features can be grouped in a subfolder if they describe a common object. The grouped filenames should represent the action and context that is covered within.

Feature FilesEvery *.feature file consists in a single feature, focused on the business value.
Gherkin TemplateFeature: Title (one line describing the story)

Narrative Description: As a [role], I want [feature], so that I [benefit]

Scenario: Title (acceptance criteria of user story)
Given [context]
And [some more context]...
When [event]
Then [outcome]
And [another outcome]...

Scenario: ...
BackgroundThe background needs to be used wisely. If you use the same steps at the beginning of all scenarios of a feature, put them into the feature’s background scenario. The background steps are run before each scenario.
ScenariosKeep each scenario independent. The scenarios should run independently, without any dependencies on other scenarios.
Scenario OutlineIf you identify the need to use a scenario outline, take a step back and ask the following question: Is it necessary to repeat this scenario ‘x’ amount of times just to exercise the different combination of data? In most cases, one time is enough for UI level testing.
Write Declarative Scenarios, Not ImperativeThe declarative style describes behavior at a higher level, which I think improves the readability of the feature by abstracting out the implementation details of the application.

Example: Imperative

Scenario: User logs in
Given I am on the homepage
When I click on the "Login" button
And I fill in the "Email" field with "me@example.com"
And I fill in the "Password" field with "secret"
And I click on "Submit"
Then I should see "Welcome to the app, John Doe"

Example: Declarative

Scenario: User logs in
Given I am on the homepage
When I log in
Then I should see a login notification

Just avoid unnecessary details in your scenarios.
Given, When, and Then StatementsI’ve often seen people writing the Gherkin syntax confuse when to put the verification step in the Given, When, Then sequence. Each statement has a purpose.
  • Given is the pre-condition to put the system into a known state before the user starts interacting with the application

  • When describes the key action the user performs

  • Then is observing the expected outcome

Just remember the ‘then’ step is an acceptance criteria of the story.
TaggingSince tags on features are inherited by scenarios, please don’t be redundant by including the same tags on scenarios.