Top 5 Quality Disasters (or Misses) of 2015

February 8th, 2016 by Ashley Hunsberger

2015 was quite the year for quality in almost every industry. Here are some defects (some disastrous, some just funny) that really caught my attention over the last year, and a few lessons we can learn from them as we develop our own test strategies such as data usage, environments, security, and more in our day-to-day work.

#1 – Social Media: Facebook tells me I’ve known you since before I was born!

Image Source:

Image Source:

This was probably one of the funnier, and definitely not critical, bugs of the year. (Let’s face it — no one actually got hurt, and I think we all chuckled a little when we saw it pop up in our news feeds!) While no one actually _confirmed_ what happened, most theories surround the Unix Epoch (January 1, 1970, which with time zone adjustments could be sometime on December 31, 1969), so 46 years from the day we started seeing our nearly golden friendships appear. Microsoft engineer Mark Davis offered his hypothesis (see

What can we learn from this? Let’s say you’re adding in a feature that has to deal with dates. Always consider testing not just with fresh data, but also data that was present prior to the feature being added. I must admit that in my own testing we hate dealing with date/time stamps as it usually involves manipulating data in unrealistic scenarios, but test we must. Make sure you are doing some time travel with your data — it helps get you closer to what your customers will have! (more…)

Paired Testing: Two Is Better Than One

January 27th, 2016 by Ashley Hunsberger

Paired programming brings two developers together to produce higher quality code compared to those same two engineers coding separately. Just as paired programming has someone writing code while another person reviews the code as it is being written, paired testing has someone doing the testing while another person takes notes, asks questions, and spots/reports bugs. I’ve personally found paired testing to foster creativity, maintain focus, provide a new way to teach others, and help release better software in general. Two testers are better than one.

Pick a partner!

It’s probably important to note that not all people actually like to pair up. Let’s face it, many of us work in a world of introverts. Some people just don’t like to talk, or share their personal space. That said, there are a few good pairings you can look at: (more…)

Changing Development Culture to Become Quality Focused

January 25th, 2016 by Joe Nolan

How many project teams have you worked on where the accepted culture was to rely on the QA members to bear the load for quality? As the leader of a QA meetup, I still constantly hear stories from my members about developers’ assumptions that it is QA’s responsibility to find bugs. Not only is this attitude demoralizing for QA, it is also not in the best interest of the team. How can a team change development culture to one that is quality focused for the entire team?

From the Top Down

The answer seems obvious — Management needs to declare that all team members will have a hand in quality. This truly is critical to a successful culture change! Besides empowering dev managers and Scrum Masters to direct team activities (such as enforcing unit tests), it will give QA members the confidence to push the team as well. (more…)

Consider Your Application’s Home: Designing for Resiliency

January 21st, 2016 by Eric Jeanes

I am a firm believer in taking a cross-discipline-based approach to technology — taking something learned in one subject area and applying it to a problem in our everyday work. The political philosopher John Rawls, in his seminal work A Theory of Justice, provides a construct that (surprisingly) has a place in specific stages of application development.

When building systems, we are constantly held back to some degree by technical debt (the time wasted by repetitive tasks, and the bug fixes required to keep systems up and running). Not only is this time costly, but it is also typically less interesting than designing and constructing new systems (naturally). (more…)

A Functional Tester Looks at Performance

January 15th, 2016 by Ashley Hunsberger

Even if you aren’t directly responsible for performance, it is important to consider it under the umbrella of quality. As a tester, how do you move forward and help drive performance quality (especially when you are new to this area, like me)? What are the ramifications of not considering performance within QA? Let’s take a look at what performance is, the questions QA can ask during design and implementation, some of the types of testing that can be done, and making performance part of your acceptance criteria (and, therefore, part of your Definition of Done).

What is software performance, and why is it important?

As an end user, I think of performance as just how fast or stable something is. If I click on something, does it take forever to load in a website? Does my app crash every time I try to open it or submit something? Do I give up and find a better solution to meet my needs? Of course we want a feature to work, but do we think about the system holistically?

I can tell you now that if a website or app that I am using crashes, I instantly think that the quality is just not there. If I have a choice in what I use, I quickly delete it and find another that does work. You may be tied into an app and not have a choice, but your opinion of that app (and the company) can quickly plummet based on stability alone.

Although performance is multi-faceted, some basic topics to think about include:

  • Response time – How quickly does the system react to user input?
  • Throughput – How much can the system accomplish in a specified amount of time?
  • Scalability – Can the system increase throughput under an increased load when resources are added?
  • Availability – Is the system available for service when requested?

To retain customers, you must consider performance as part of overall quality.

Understanding performance during development

The problem I’ve seen is that performance is always deemed important, but is not necessarily addressed up front. All too often I recall discussing performance long after a feature was coded and tested. It was pushed until the end, and it can be difficult to make your features meet performance expectations after they’ve been built. This was difficult in a Waterfall world, but how do you adapt what was an afterthought as more and more companies are moving towards a Continuous model? Performance needs to be considered first and understood by the team.

Here are some sample questions to be considered DURING design and development to help ensure you are considering performance needs early on as you go through your non-functional requirements (remember, you will need to discuss as a team and get guidance in determining what is expected):

CategoryNon-Functional Requirement Questions
Response TimeWhat is the acceptable waiting time for users?

Do we need to consider users on various devices and speeds? Do we need to simulate slower speeds? Some may be on modern desktop/laptops on high speed Internet, or modern mobile devices on 3g and above — but others may not.

Example - Changing a password. How long before I can expect a change to take effect? Do I need to show progress feedback?
Data VolumeHow do we ensure that data volume does not impact user experience?

What's the maximum and typical volume of data that will be involved?

Example - Entering a page that lists users. Do I show all 20,000 users in the system? Do I show the first 25? How long does it take? Can I perform other actions while the list generates? Do I see a blank screen while I wait?
CachingCache is king. Queried and calculated data can be reused to eliminate duplicate work.

When do we need to invalidate the cache? Which data cannot tolerate staleness?

How long can the cache live for?

Could cache staleness impact the system or the user session only?

Example - Notification badges. How long do those notifications last once a user has viewed, or just upon first login? If the cache is stale, does it impact just the authenticated user?

Testing performance

There are several types of testing that can help ensure your apps are performing as expected. Please note that this is not a comprehensive list, just a high-level overview to get you started1)Summarized from and

Type of TestOverview
LoadApplication is tested for response times during normal and peak usage. How does the app respond with a few users completing a few interactions vs. thousands of users completing thousands of interactions at a time?
StressFinds ways to break the system by increasing the load. Start with a good benchmark (identified during your load testing), and increase the load until you see which components start lagging and fail first.
VolumeTest if application performance degrades with more data volume. Do you access the database directly? How does it handle the query if there are millions of records?
Reliability/RecoveryIf your app does fail, testing will show if and how it recovers, and how long it takes to get back to an acceptable state.
ScalabilityTests if your app’s performance improves if you add resources (hardware, memory, etc.)

Improving performance quality faster

It’s time to stop pushing performance to the end and hoping for the best. As stories are designed, add performance to your acceptance criteria. Make sure that everything in your acceptance criteria is marked as complete (part of your Definition of Done).

As with anything, the longer you put something off, the more difficult (and/or expensive) it is to implement later. Be proactive, and build performance in.

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices.  Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen.  In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

References   [ + ]

1. Summarized from and

Planning Quality Architecture for 2020

January 11th, 2016 by Greg Sypolt

I was inspired by Denali Lumma (@denalilumma) when she delivered a glimpse of the future in her talk about 2020 testing at the Selenium 2015 conference. The session was an excellent introduction that compared many scenarios of the minority testing elite versus the more common development team. The elite companies consider infrastructure FIRST, and the majority thinks about infrastructure LAST. It got my wheels turning regarding the future of software development. I don’t have all the answers right now, but I want to be part of the movement to plan and build architecture with quality. A few words come to mind when thinking about quality architecture — automation, scalability, recoverability, and analytics.

Build a culture

When building a culture, avoid too much control. You want a culture that embraces freedom, responsibility, and accountability. Why is building a culture like this important? It allows passionate employees to innovate and find big-time solutions. You can’t plan for innovation. It naturally happens. When you give passionate employees an inch, they’ll take a mile. The future team culture needs to push the envelope and step outside their comfort zone. (more…)

Re-Energize Your QA Career With Automation and DevOps

January 7th, 2016 by Joe Nolan

It’s time for you to stop being content with the status quo and re-energize your QA career with Automation and DevOps — otherwise, you might find yourself fading away like Marty McFly! I’m talking to YOU, manual tester! And YOU, QA manager! Oh, and YOU TOO, automation engineer! Every one of you who has a vested interest in your career growth needs to familiarize yourself with automation and DevOps tools.

Of Course You Need to Understand Automation

Let’s face it: In this day and age of software development, speed is the key to survival. In order to achieve clean builds, Continuous Integration, Continuous Delivery, and Agile development, manual testing just ain’t gonna cut it.

Everyone with the QA title needs to continuously build on their skill set, just like a developer. Even if you aren’t actively writing automation code, you still need to understand the capabilities and benefits of each type of automated test, especially the ones written by your development team. The team is relying on your expertise to guide them with acceptance criteria for stories, while bringing QA concepts to the table. (more…)

The Importance of a Triage Team

December 30th, 2015 by Joe Nolan

I grew up watching shows like M*A*S*H, and Emergency!. Doctor and paramedic characters would perform triage of injuries and determine which ones were critical and which could wait. If you think about it, bugs in a feature are like injuries to your code, and when they are discovered, they too need to be triaged. Without triage, bug tickets can add time to your development process and even cause invalid fixes. Every development team should triage their bugs!

Just What Does a Triage Team Do on a Development Team?

How many times have you worked on a bug that says something like “this feature is broken”? You might think this is an exaggeration, but it’s really not too far off. Especially if your team conducts bug bashes with users who don’t normally write tickets. This will either start a round-and-round process of different team members clarifying the ticket, or worse, a developer will take it upon him or herself to fix what he or she THINKS is implied. All of this is a time suck to the team.

On the other hand, how many times have you wondered why some tickets are being worked while more critical tickets are just sitting there? How frustrating is that? This happens frequently if tickets are not prioritized and put into the backlog.

A good triage team prevents all of this!


Accessibility (AX) Testing in the DevOps Chain

December 28th, 2015 by Ashley Hunsberger
Image Source:

Image Source:

We often hear how important accessibility (AX) is for delivery, but we consistently see that it’s one of the first things cut as we get closer and closer to release (and we hope we’ll have the chance to properly address it later). As more companies embrace DevOps and Continuous models, how can we ensure that this extremely important aspect of development is no longer an afterthought?

Ensuring accessibility takes understanding of what AX is, asking the right questions during design and implementation, knowing the tools that are available to help you, and making accessibility part of your acceptance criteria (and therefore, making it part of your Definition of Done).

What is Accessibility?

Web Accessibility allows people with diverse abilities to use the Web1) — for example, people with cognitive issues (such as autism and ADHD) or physical (blindness, auditory, etc.). I recommend reading Anne Gibson’s post. She provides insight on a wide variety of health issues which many people might never consider — that could happen to any of us at any time — and the user personae. (more…)

References   [ + ]

Balancing Documentation With Speed

December 23rd, 2015 by Ashley Hunsberger

I’m sitting here thinking about my career over the last decade or so, thinking about how things once were, and how things have changed. One thing stands out to me: how much faster things are now. How fast we build, how fast we release, how quickly we have to test, how quickly we have to write. How do we balance documentation (which, let’s face it, could take a lot of time) with the need to get things out the door? How do we keep up in the continuous pipeline? It comes down to having a centralized strategy that consists of best practices, how to get through your day-to-day work, and working as a team to know what needs to be tested.

One Strategy to Rule Them All

One thing I’ve found that has helped to save time is to have a centralized space to reference when nothing really changes in HOW things need to be tested. These are things that do not change across projects or releases and should be considered as long-term and used as a reference. We used to do this for each epic and it drove me nuts — it ultimately led to tons of strategies with duplicate information throughout. Now, we have a reference wiki space and go over strategies that everyone (not just QA) should be familiar with. These strategies should also be a part of passing acceptance criteria, and your engineers need to know these strategies as well. (more…)