Using QA to Enhance Communication

July 21st, 2015 by Ashley Hunsberger

Have you ever worked on a project and found yourself constantly shaking your head? I can say that 99% of the time that I experienced frustration, it was largely due to communication issues within a team. I’ve personally been on project teams and wondered if anyone there had ever taken a basic communications course and learned concepts like active listening, empathy, and being clear and concise. A team that can communicate will find success, but what about those who are not interacting well? Who can help your team get back on track? Believe it or not, your answer is the tester.

The Communication Breakdown

What exactly is keeping your team from communicating effectively? The answer may not be so obvious. In my experience, I’ve seen a few key contributors that include, but are not limited to:

The team does not own quality
Of course life would be easier if we could say ‘Well, I did my job!  I’m done! Time for the tester!”  But how does that foster communication in a team if you just throw something over the proverbial fence? If you set out only to do your job and not to understand anyone else’s, are you really being part of a team? You know the saying — “There is no I in TEAM.” We usually hear that as kids when we first embark in sports, but it goes for software development projects, too!

Not seeing the big picture
It is so easy to start looking in an Agile world at the small, granular user story level.  But when you start to lose the big picture, the ability to communicate becomes more and more difficult. There have been projects I’ve worked on where I just didn’t understand the business reasons (and I wasn’t the only one). Can you guess how many of those projects succeeded?

Not knowing exactly what to build
So many times, I’ve seen (and worked on) projects where a designer says “build this.” The engineer goes off and starts building, and the tester starts writing tests — presumably for the same feature.  But if an engineer has a clarifying question and does not include the tester when asking the designer, or vice versa, then what?  It’s an easy recipe that gets off track when you work in a silo.

Enter the Tester

Your tester can help alleviate many communication gaps. It just requires a different way of thinking.  Take each of the breakdown factors I’ve mentioned, and let’s look at how this QA team member is uniquely positioned to enhance your team’s communication and collaboration.

Everyone owns quality! Yes, you too, engineers…
Rather than creating a team that thinks QA alone is responsible for the quality of a product, what if everyone had skin in the game? Of course, part of that is understanding each other’s roles and what the value of testing is. A good tester will work with your team to decide what types of testing need to occur, who can take care of them, and free up those people to take on testing activities. One way to understand this is to take a look at the Agile Testing Quadrant.

agility_test_quadrants

If everyone is concerned about quality, and everyone starts talking about quality, and they aren’t just checking out as soon as their task is ‘complete’, you’re already on the path to improved communication.  You take it to a whole new level when you get people INVOLVED in quality.

The 30,000 foot view
I mentioned before that  it’s very easy to get caught in the weeds looking at an individual user story.  A tester should, however, know where that user story falls in the grand scheme of things — understanding the business value a feature brings, as well as the impacts it may have on other features.  With teams I have worked with in the past, we have invested time in making sure any new QA team member learns the product — not just the one area they are working on. (To repeat,  not just be the tester with the product knowledge.) It will be beneficial to anyone on the team, but your tester can help drive the conversations needed to understand the user story, taking a step back to see where it all fits in, and guide testing in a direction perhaps maybe not thought of before.

Everyone on the same page
Let’s imagine a world where we all go about building and testing the same product, with the same vision.  Is that possible when you work isolated from your team? Let’s look at a central testing practice for Agile teams called Specification by Example (also known as Acceptance Test-Driven Development, or ATDD).  The key idea here is that your tester, engineer, and designer all work together BEFORE anything is built to understand the feature, write the tests (your acceptance criteria), and know exactly how the feature will be tested. It is done when all tests pass. Manual or automated, this practice gets everyone on the same page. Your tester can help drive these conversations (I have often gone in with a list of tests, but typically just to elicit more questions and answers with the engineers and designers to get clarity and walk away with more tests — or changed tests, and ensure that we all understand the acceptance criteria as we have now written the tests together).

What are you waiting for?
I hope these concepts get you going in the right direction. Use your tester! QA is not just there to execute a test, but is uniquely structured to help foster communication and collaboration on your team, and make sure everyone is invested in quality.

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices.  Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen.  In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

T: @aahunsberger
L: https://www.linkedin.com/in/ashleyhunsberger

 

Don’t Let Them Leapfrog You

July 15th, 2015 by Ashley Hunsberger

The entry barrier for nearly all markets has been dramatically reduced, as continuous integration and delivery allows very small companies to leapfrog massive institutions. It’s no longer just about having that extra feature to get clients, it’s also about how you deliver. Let’s see why companies that have adopted continuous delivery are leaving their competitors in the dust — and how you can, too.

A Tale of Two Companies

“It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way.” 
Charles Dickens: A Tale Of Two Cities (1859)

You may ask what on Earth one of my favorite novels has to do with testing and continuous delivery, but stay with me for a minute (or five). No, I won’t go into the French Revolution (though moving to continuous integration may feel like you are sparking a revolution in your development team). Let’s take a look at two businesses.

Meet Company A. They’ve been around awhile. In fact, they were one of the first companies in their market. As with most older companies, the company began as a waterfall shop and slowly transitioned to Agile, but is still showing signs of waterfall (something I like to call Scrummerfall).  They still get specs, they develop, and they test,  albeit in defined sprint time frames. Tests may be written while development is going on, but are performed after development hands off to QA.

Now, meet Company B, Company A’s competitor. Small and lean, the team has adopted continuous delivery, and acceptance test driven development (ATDD). They write tests first, and develop until all tests pass. They build quality in, rather than check for it later.

And the Winner Is…

Can you guess which company is chipping away at the market? The clear winner here is Company B.  Embracing DevOps has allowed Company B to realize several benefits, including (but not limited to):

Faster time to market – Get a desired product into customers’ hands faster!

Better quality product – Build the quality in! Find the bugs before QA even gets the product, or even more importantly, before customers see it.

Reduced cost of development – The cost to find bugs early (write tests first, build until all tests pass) is MUCH lower than the cost of finding them later in traditional testing cycles. And if a bug is found by the client? The cost skyrockets.  Build the quality in and reduce costs down the line.

Higher customer satisfaction – Continuous integration (CI) yields faster delivery, which means faster feedback. Issues are fixed faster, which means products are in clients’ hands faster. The customer has trust in your company and product.

ibm_blog_image.

Source: IBM http://www.ibm.com/developerworks/devops/

A Cultural Shift

Getting your team on the road to continuous delivery is not just a process change. It is a huge cultural shift. It is a completely different way of thinking, and it does not lie only within the development team. How will Sales present the company? How will Marketing develop materials? How will Support handle the change in release cadence? How will product managers adapt to learn from customers? It’s a ripple effect that will impact the entire company, for the better.

The longer you take to embrace DevOps (and continuous delivery), the more your competition has a chance to move in. Take your company to the next level. Be the change!

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices.  Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen.  In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

T: @aahunsberger
L: https://www.linkedin.com/in/ashleyhunsberger

An Efficient Release Made is a Penny Earned

July 9th, 2015 by Greg Sypolt

Introduction

Everyone wants to save money and deliver faster, right? Even so, you would be surprised how many software development organizations still do not practice continuous integration. They still depend heavily on expensive manual testing. Is your software development organization ready to change, to deliver faster with built-in quality and fewer meetings to boost productivity?

Without analysis, it is hard to realize the ROI (return on investment) for CI (continuous integration) and automated tests. Like any type of change, it requires analysis of investment cost, the timeline, and ROI before it’s adopted into your development organization.

Building In Quality

There is one common trend in software development organization — ‘building quality in’ at the beginning rather than considering quality at the end of the process.

EVERYONE owns quality — yes, even the product designers and developers. Everyone has visibility for all types of tests upfront (unit, integration, e2e, and manual).

Why should engineers and designers be involved? They help us understand what is possible at the unit, business, and front-end layers. By having everyone involved, the team can ALL agree on the acceptance criteria.

By including the developers and designers in your test strategy, everyone owns quality. You are there to drive that quality — not just be a tester. Engineers are testers. Designers are testers. Everyone can contribute!

Building quality in will prevent bugs rather than lead to the discovery of bugs at the end of the development cycle. The effect of building quality in therefore raises your ROI, and here’s how:

  • Built-in quality, reducing the number of bugs — since you are now preventing bugs, not just finding them.
  • Reduced manual testing, because testing has become automated, and most importantly, you’ve developed the right tests.
  • Quicker and more confident releases
  • Stabilized product that may give product management a chance to breathe and redefine the product strategy.
  • It is more fun for developers to work on a system that behaves predictably, doesn’t break anytime anyone touches it (and if it does break, you know immediately, not a week down the road).

Building quality in can make your system behave predictably. Slow adoption of continuous integration and automated tests makes it hard to realize ROI in this process. The most common (and hidden) cost in manual testing is finding a bug. So how do these effects translate into a dollar value?

PennyEarned_v2

IBM Research presented at AccessU Summit 2015

Here are perfect examples of the potential for improvement in ROI when ‘building quality in’ after adopting CI and automated tests without immediately seeing the savings:

A developer commits some bad code which breaks the build, and sometimes cripples the entire manual test for hours or days. Adopting continuous integration and automated tests will not cripple your QA teams.

Manual testing takes several days to complete, along with reporting the test results. Manual testing is repetitive (and kind of boring) after performing the same tests multiple times. Human errors result from manual testing. Adopting automated tests allows testing that’s repeatable and provides testing results within minutes, instead of waiting for days or weeks for testing results.

Wasting Time in Meetings

A simple way to boost productivity is to reduce meetings and stop wasting time in meetings by creating simple guidelines.

Let’s acknowledge that meetings do have a place in our development lives. They are great tools for planning, sharing information, and quickly getting everyone on the same page. Then there are those recurring meetings, often scheduled for the sake of having a meeting. At those times, it doesn’t hurt to question why you need to be present.

Example:
You have five employees meeting in a conference room or virtually. Let’s say the average annual salary is $55K and the company’s recurring meeting is once per week for one hour. How much does this meeting cost your company? Approximately $6,500 per year.

Time spent together is not always time spent getting work done. It is important to create simple guidelines to eliminate wasteful meetings and save the company money. Here’s some suggestions to help make meetings more productive:

  • Spend more time on the agenda and allow the attendees time to review.
  • Invite the right people — Don’t waste people’s time.
  • Always start the meeting on time, regardless of who may be late.
  • Avoid one-hour meetings unless they are completely necessary. You can even opt to switch that weekly one-hour meeting to a 10-15 minute stand-up meeting.
  • End the meeting at the agreed-upon time, even if the agenda is not finished.
  • Table any discussion that is not relevant to the agenda.

Conclusion

By following some of these suggested guidelines, reducing the number of meetings, and working smarter, your company will start to see real ROI. An efficient release made is a penny earned.

Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. He is a passionate automation engineer seeking to optimize software development quality, coaching team members how to write great automation scripts, and helping testing community become better testers. Greg has spent most of his career working on software quality – concentrating on web browsers, APIs, and mobile. For the past 5 years he has focused on the creation and deployment of automated test strategies, frameworks, tools, and platforms.

It worked on my machine – Communication Trap

July 1st, 2015 by Ashley Hunsberger

“I don’t see it on my machine!” said every developer ever. Every QA professional I have talked to in my career has heard this at least once.

But why?

Have we asked what’s in a bug?

The answer can either be what gets your team on the road to efficiency, or it can become a kink in the delivery hose. Let’s discuss how your QA can help the team deliver faster by providing a consistent language to keep everyone on target.

Don’t Let The Bad Bugs Bite…

Over the last decade, I have seen issues that have almost no noted content in them (seriously, some have just declared something to the tune of “This feature is… not working”). Then there are tickets that are the golden standard, that have all the information you could possibly want (and probably some with more than you need that turn out to be a few bugs in themselves).

But what happens when you don’t have a common way to report a ticket, and why is it important?

I just came across an issue recently that seemed to have some steps to reproduce, but the setup was not included. Try as I might, I could not replicate the bug. The only way that I could come close to the reported result did not match the steps provided, and I could only guess that the setup I created was what the reporter had done. I will let you guess how long this issue took. Hint: It wasn’t a few hours.

Or perhaps you have an offshore team. I’ve seen many, many instances when someone reports a bug that just doesn’t have enough information in it. If the engineer cannot figure out exactly what the issue is, and has to place it on hold, back to the reporter, the engineer waits another night while the person on the other side of the world hopefully notices the ticket is back in his or her queue for more details. That is another full day that the bug exists, delaying when the root cause can be identified and the issue fixed.

Depending on the makeup of your team, and whether you are in an automated or manual setup — you need to consider how the issue will be verified. The person testing the fix (or writing the automated test to ensure the issue does not occur again) may not be the one who reported it. (Again, more time is spent figuring out how to test if the fix is correct.)

The bottom line? The back and forth that occurs from a poorly reported bug is costly in terms of time and resources.

Cut The Chit Chat

Having a uniform language/template will help reduce uncertainty across the board, and reduce the time a bug is spent unresolved. But what should be included in a bug report to cut out this back and forth, and keep the team on track?  There are several other things you may want to consider adding, but these are some of the top things I like to see from a tester:

  • Summary/Title: This should be succinct yet descriptive. I almost try to make these sound like a user story <user> <can/cannot><do x action> in <y feature>. When I sit in a triage meeting, can I tell what the issue is just by reading the summary?
  • Environment: every now and then we come across bugs that are very specific to the OS, database type, browser, etc.  Without listing this information, it’s all too easy to say ‘Can’t reproduce’, only to have a client find it in the field.
  • Build: Hopefully you are testing on the latest build, but if for some reason you have servers that are updated at different rates than others, you need to pinpoint when exactly the bug was found.
  • Devices: if you’re doing any type of mobile testing, what type of device were you using? What version? If you found a bug on the web app, do you see it on the mobile app too? Which one? Android or iOS?
  • Priority: The priorities are all relatively standard across the field — Critical, High, Medium and Low. Have criteria defined up front so everyone is on the same page as to what constitutes each selection.
  • Steps to reproduce: Not just ‘When I did this, it broke.’  Really break it down, from login and data setup to every click you make.
  • Expected Result vs. Actual Result: What were you expecting, and why?  What happened instead?
  • Requirements and Wireframes: This helps to point to why testing occurred, and why someone wrote up a bug and linked it back to the originating artifact, though hopefully you are on the same page upfront, before development begins. Sometimes things slip through and perhaps an engineer has a different understanding of a feature than the tester. Being able to point back to why you think an element is a bug is helpful, and gets you all on the same page.

Of course, there are people other than your traditional testers writing bugs, and it is essential to use your QA to drive conformity. Perhaps your UX team is performing audits, or you have bug bashes where people from other departments are invited to test the system and find bugs, or you have someone new to the team that simply needs training. Having a template will ensure clarity and reduce inefficiencies, regardless of who enters the ticket.

Utilize QA to promote consistency, get bugs out of purgatory, and drive faster delivery.

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices.  Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen.  In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

T: @aahunsberger
L: https://www.linkedin.com/in/ashleyhunsberger

Guest post: Proving that an application is as broken as intended

June 25th, 2015 by Björn Kimminich
Typically you want to use end-to-end (e2e) tests to prove that everything works as intended in a realistic environment. In the Juice Shop application that idea is changed to the contrary. Here the main purpose of the e2e test suite is to prove that the application is as broken as intended!

Juice Shop: Broken beyond hope – but on purpose!

“WTF?” you might ask, and rightfully so. Juice Shop is a special kind of application. It is an intentionally insecure Javascript web application designed to be used during security trainings, classes, workshops or awareness demos. It contains over 25 vulnerabilities that an aspiring hacker can exploit in order to fulfill challenges that are tracked on a scoreboard.

The job of the e2e test suite is twofold:

  1. It ensures that the overall functionality (e.g., logging in, placing products in the basket, submitting an order, etc.) of the application is working. This is the above mentioned typical use case for e2e tests.
  2. It performs attacks on the application that should solve all the existing challenges. This includes SQL Injections, Cross-Site Scripting) attacks, business logic error exploits and many more.

 

When does Juice Shop pass its e2e test suite? When it is working fine for the average nice user and all challenges are solvable, so an attacker can get a 100% on the scoreboard!

Juice Shop logo

Application Architecture

Juice Shop is created entirely in Javascript, with a Single-Page-Application frontend (using AngularJS with Bootstrap) and a RESTful backend using Express on top of NodeJS.

 

The underlying database is a simple file-based SQLite with Sequelize as a OR-mapper and sequelize-restful to generate the simple (but not necessarily secure) parts of the API dynamically.

Test Stages

There three different types of of tests to make sure Juice Shop is not released in an unintendedly broken state:

  1. Unit tests make sure that the frontend services and controllers work how they should. The AngularJS services/controller are tested with Karma and Jasmine.
  2. API tests verify the RESTful backend is behaving properly when running as a real server. These tests are done with Karma and frisby.js for orchestrating the API calls.
  3. The e2e test suite performs typical use cases and all kinds of attacks via browser-automation using Protractor and Jasmine.

 

If all stages pass and the application survives a quick monkey-test by yours truly it will be released on GitHub and SourceForge.

Why Sauce Labs?

There are two reasons to run Juice Shop tests on Sauce Labs:

  1. Seeing the front-end unit tests pass on a laptop already gives a good feeling for an upcoming release. But there they run only on PhantomJS, so not in a real browser. Seeing them pass on various browsers increases confidence in the release.
  2. The e2e tests must be executed before shipping a release. Wanting to make sure they are not skipped due to laziness or overconfidence (“Oh’ it’s such a small fix, what could it possibly break?” – sound familiar?) the e2e suite must be integrated into the CI pipeline.

 

Having laid out the context the rest of the article will explain how both these goals could be achieved by integrating with Sauce Labs.

Execution via Travis-CI

Juice Shop builds on Travis-CI which Sauce Labs integrates nicely with out of the box. The following snippet from the .travis.yml shows the necessary configuration
and the two commands being called to excecute unit and e2e tests.

addons:
  sauce_connect: true
after_success:
- karma start karma.conf-ci.js
- node test/e2eTests.js
env:
  global:
  - secure: <your encrypted SAUCE_USERNAME>
  - secure: <your encrypted SAUCE_ACCESS_KEY>

Frontend Unit Tests

The karma.conf-ci.js contains the configuration for the frontend unit tests. Juice Shop uses six different OS/Browser configurations:

var customLaunchers = {
    sl_chrome: {
        base: 'SauceLabs',
        browserName: 'chrome',
        platform : 'Linux',
        version: '37'
    },
    sl_firefox: {
        base: 'SauceLabs',
        browserName: 'firefox',
        platform: 'Linux',
        version: '33'
    },
    sl_ie_11: {
        base: 'SauceLabs',
        browserName: 'internet explorer',
        platform: 'Windows 8.1',
        version: '11'
    },
    sl_ie_10: {
        base: 'SauceLabs',
        browserName: 'internet explorer',
        platform: 'Windows 8',
        version: '10'
    },
    sl_ie_9: {
        base: 'SauceLabs',
        browserName: 'internet explorer',
        platform: 'Windows 7',
        version: '9'
    },
    sl_safari: {
        base: 'SauceLabs',
        browserName: 'safari',
        platform: 'OS X 10.9',
        version: '7'
    }
};

 

In order associate the test executions with the Travis-CI build that triggered them, some extra configuration is necessary:

    sauceLabs: {
        testName: 'Juice-Shop Unit Tests (Karma)',
        username: process.env.SAUCE_USERNAME,
        accessKey: process.env.SAUCE_ACCESS_KEY,
        connectOptions: {
            tunnelIdentifier: process.env.TRAVIS_JOB_NUMBER,
            port: 4446
        },
        build: process.env.TRAVIS_BUILD_NUMBER,
        tags: [process.env.TRAVIS_BRANCH, process.env.TRAVIS_BUILD_NUMBER, 'unit'],
        recordScreenshots: false
    }
    reporters: ['dots', 'saucelabs']

 

Thanks to the existing karma-sauce-launcher module the tests are executed and their result is reported back to Sauce Labs out of the box. Nice. The e2e suite was a tougher nut to crack.

End-to-end Tests

For the Protractor e2e tests there are no separate configuration files for local and CI, just one protractor.conf.js with some extra settings then running on Travis-CI to pass necessary data to Sauce Labs:

if (process.env.TRAVIS_BUILD_NUMBER) {
    exports.config.seleniumAddress = 'http://localhost:4445/wd/hub';
    exports.config.capabilities = {
        'name': 'Juice-Shop e2e Tests (Protractor)',
        'browserName': 'chrome',
        'platform': 'Windows 7',
        'screen-resolution': '1920x1200',
        'username': process.env.SAUCE_USERNAME,
        'accessKey': process.env.SAUCE_ACCESS_KEY,
        'tunnel-identifier': process.env.TRAVIS_JOB_NUMBER,
        'build': process.env.TRAVIS_BUILD_NUMBER,
        'tags': [process.env.TRAVIS_BRANCH, process.env.TRAVIS_BUILD_NUMBER, 'e2e']
    };
}

 

The e2e tests are launched via e2eTests.js which spawns a separate process for Protractor after launching the Juice Shop server:

var spawn = require('win-spawn'),
    SauceLabs = require('saucelabs'),
    colors = require('colors/safe'),
    server = require('./../server.js');

server.start({ port: 3000 }, function () {
    var protractor = spawn('protractor', [ 'protractor.conf.js' ]);

    function logToConsole(data) {
        console.log(String(data));
    }

    protractor.stdout.on('data', logToConsole);
    protractor.stderr.on('data', logToConsole);

    protractor.on('exit', function (exitCode) {
        console.log('Protractor exited with code ' + exitCode + ' (' + (exitCode === 0 ? colors.green('SUCCESS') : colors.red('FAILED')) + ')');
        if (process.env.TRAVIS_BUILD_NUMBER && process.env.SAUCE_USERNAME && process.env.SAUCE_ACCESS_KEY) {
            setSaucelabJobResult(exitCode);
        } else {
            server.close(exitCode);
        }
    });
});

 

The interesting part regarding Sauce Labs is the call to setSaucelabJobResult(exitCode) in case the test is run on Travis-CI with Sauce Labs credentials which are passed in by the extra config part in protractor.conf.js.

This function passes the test result from Protractor on to Sauce Lab’s REST API:

function setSaucelabJobResult(exitCode) {
    var sauceLabs = new SauceLabs({ username: process.env.SAUCE_USERNAME, password: process.env.SAUCE_ACCESS_KEY });
    sauceLabs.getJobs(function (err, jobs) {
        for (var j in jobs) {
            if (jobs.hasOwnProperty(j)) {
                sauceLabs.showJob(jobs[j].id, function (err, job) {
                    var tags = job.tags;
                    if (tags.indexOf(process.env.TRAVIS_BUILD_NUMBER) > -1 && tags.indexOf('e2e') > -1) {
                        sauceLabs.updateJob(job.id, { passed : exitCode === 0 }, function(err, res) {
                            console.log('Marked job ' + job.id + ' for build #' + process.env.TRAVIS_BUILD_NUMBER + ' as ' + (exitCode === 0 ? colors.green('PASSED') : colors.red('FAILED')) + '.');
                            server.close(exitCode);
                        });
                    }
                });
            }
        }
    });
}

 

This was necessary because there was no launcher available at the time that would do this out-of-the-box.

Determining solved Challenges

How does Protractor get its test result in the first place? It must be able to determine if all challenges were solved on the score board and cannot access the database directly to do that. But: It can access the score board in the application:

Screenshot score board

As solved challenges are highlighted green instead of red some simple generic function was used to assert this:

protractor.expect = {
    challengeSolved: function (context) {
        describe("(shared)", function () {

            beforeEach(function () {
                browser.get('/#/score-board');
            });

            it("challenge '" + context.challenge + "' should be solved on score board", function () {
                expect(element(by.id(context.challenge + '.solved')).getAttribute('class')).not.toMatch('ng-hide');
                expect(element(by.id(context.challenge + '.notSolved')).getAttribute('class')).toMatch('ng-hide');
            });

        });
    }
}

 

When watching the e2e suite run Protractor will constantly visit the score board to check each challenge. This is quite interesting to watch as the progress bar on top moves closer to 100% with every test. But be warned: If you plan on trying to hack away on Juice Shop to solve all the challenges yourself, you will find the following screencast to be quite a spoiler! ;-)

Bjoern Kimminich is responsible for IT architecture and application security at Kuehne + Nagel and as a side job is giving lectures on Software Engineering at the private university Nordakademie. When not working on his ownJuice Shop, Bjoern thinks up Code Katas and regularly speaks at conferences and meetups on topics like application security and software craftsmanship. Twitter: @bkimminich

Immutable Infrastructure

June 23rd, 2015 by Greg Sypolt

Server hugging is a disease

In the dark days before immutable servers, people clung to servers and treated them as untouchable gold. These people still exist, and hang onto their servers instead of moving into the Cloud. They are server huggers. What does the term “server hugger” mean? Its the desire to “touch” servers, “reboot” them on a regular basis, constantly upgrading really old software and hardware needed, etc. Before anyone can help cure the problem, server huggers have to admit it. They struggle to let go and embrace the Cloud. The thought of trashing servers and disposable components is absolutely frightening to server huggers, especially the (unfounded) fear of losing control. Server virtualization and Cloud computing will make these types extinct in the near future. Let’s start unshackling your servers!

Unshackle your servers

It’s all within reach. Start the transition today! Before the unshackling can begin, you need to build a Cloud architecture strategy for predictability, scalability, and automated recovery. The next step is huge: getting buy-in from development and operation teams. It starts by presenting your strategy and demonstrating the importance of virtual servers and containers to make it possible. Let’s start setting your dedicated servers free!

Start small and expand slowly! Always look for ways to simplify your infrastructure — build, measure, and learn. Some people like to learn by jumping directly into writing their own code and others may seek experts. Choose your own adventure! You are making giant leaps towards immutable servers. Do not treat virtual machines in a static way.

Make infrastructure part of the application

What is an immutable infrastructure? It is made up of immutable components that are replaced at every build and deployment, with changes made only by modifying a versioned definition, rather than being updated in place. So when building infrastructure as part of the application, the following three words immediately come to mind:

  • Predictable – Promote the exact same artifact that you tested into your production system.
  • Scalable – Meet user demand so the application can automatically grow or shrink your servers.
  • Recoverable – Using Auto Scaling Group (ASG) should detect the instance termination and automatically bring up a new, identically configured instance. Netflix has taken this a step further with its Chaos Monkey, a service which randomly knocks out production services, forcing developers to build easily recoverable infrastructure.

Why is this important? Without this infrastructure, you can spend hours each week manually maintaining and modifying network configurations and the problems will still continue to grow. It’s just not a smart use of employee resources.

There are many benefits and advantages to moving your infrastructure to the Cloud. Let’s start with the possible cost savings — You can pay for how much you consume, instead of having to invest heavily in data centers and servers. Next, stop guessing your infrastructure capacity and speed needs. You can access as much or as little as you need, and scale up and down as required in only a few minutes. Another plus — Where it matters the most, you can easily deploy your application in multiple regions around the world in a few minutes. This means you can provide a lower latency and better experience for your customers. To ease the management of your Cloud infrastructure, you will benefit from many available Cloud infrastructure automation tools.

When all is said and done, you can release the entire stack faster to market with greater test coverage since you’ve helped the server huggers move on.

How

The Cloud! Start turning your infrastructure into code. Enable developers to build, test and deploy applications on highly scalable and reliable infrastructure. Think about what you want the servers to run, not how to run them.

When starting the shift to immutable infrastructure, consider these steps:

  • Start small
  • Get buy-in
  • Implementation
  • Reinforce best practices and automate the deployment
  • Adopt automation across the deployment process
  • Build on your successes, so you can begin to reap the benefits of immutable servers

The takeaway:

  • Get over static servers, and treat servers as a commodity.
  • Treat all our virtual servers as immutable.
  • The deployment model is to terminate the instance/container and start over from step one: build a new image and throw old instances away.
  • Script your orchestration, and expand your ability to test more and build more comprehensive grids.
  • When designing your Cloud infrastructure from scratch, it makes sense to start small and expanded slowly — don’t try to “boil the ocean.”
  • The Cloud has made immutable infrastructure possible.

Greg Sypolt (@gregsypolt) is a senior engineer at Gannett and co-founder of Quality Element. He is a passionate automation engineer seeking to optimize software development quality, coaching team members how to write great automation scripts, and helping testing community become better testers. Greg has spent most of his career working on software quality – concentrating on web browsers, APIs, and mobile. For the past 5 years he has focused on the creation and deployment of automated test strategies, frameworks, tools, and platforms.

Moving QA Upstream

June 17th, 2015 by Ashley Hunsberger

Every few years, new development and release methods appear. Lately, most of us are looking at continuous integration and continuous delivery. But what about those teams that are trying to transition to faster releases? That’s going to be a huge hurdle if you save your testing for the end, before release (and in my experience, that has almost always resulted in pushing the release date). There is an important shift that needs to be made in order to be successful: bring in QA early.

Moving Upstream – From the Field

I recently participated in a sprint retrospective. During our ‘lessons learned’ meeting, an engineer said, “I really wish we had had these conversations with QA earlier… they really knew the feature.” In my perfect world, the referenced discussion would have happened before any development had begun. Unfortunately, sometimes factors are out of our control (like the number of resources — I have discovered that I cannot, in fact, clone myself). The end result of this meeting was that the engineer realized that some unexpected new work and some revisions to code he had already written would need to be done. Extra scope aside, the engineer brought up a great point: the tester knows the system/feature, and really does know what questions to ask.

When you bring QA in earlier and discuss a feature up front with the engineer and designer — before any work is done — all questions and assumptions are out on the table. This pre-mortem allows the team to size up possible issues, and prepare for feature testing with better expectations. Together, you eliminate the guesswork that comes from working on your own, which will likely be different depending on whether you are a developer or a tester. When QA and devs work together, engineers know precisely what they have to develop in order for the feature to pass and meet the acceptance criteria, and when it is time to audit, designers should not be surprised.

The benefits of moving QA upstream go beyond getting everyone on the same page. You will also reduce the overhead of over-documenting your project, since the acceptance tests become your specifications. There is no more need to tediously update both test and spec if you are defining them in your acceptance criteria as a team. I wish I had a dime for every time I heard, “Was it updated in the spec so I can update the test (or update the code)?” … Let’s just say I’d have a lot of dimes. Often this conversation happens in various emails or chats where someone thought the behavior was discussed, and may or may not have included all necessary team members.

To me, the biggest reward for defining acceptance criteria first is that you are building quality in from the start. Of course you’ll find bugs — I wouldn’t be where I am today if software came bug free. But you will certainly prevent bugs, and hopefully the bigger ones. Think about it, if you define acceptance criteria first, you build to the test until all tests pass — the bugs found during that process are found early so you can release faster but maintain quality. The cost alone for finding a bug late is so much higher than finding a bug early (spending resources on triage, remembering what was built a sprint or so ago while you’ve possibly moved on to another user story, potentially impacting other code with your fix).

Step 1 – A Strategy

To be truly successful with continuous integration, I think you really do need to look at automation — but the great thing is, moving QA up in your development lifecycle is completely tool and methodology agnostic. What do I mean? It doesn’t matter if you have a manual testing group or a robust automation shop. Getting your QA involved earlier can only help your team. Of course, as you write your acceptance criteria, you may want to also identify how that criteria can be tested (for example, is it a unit, integration, or functional test), but in the grand scheme of things, it doesn’t matter what your testing framework is. Hopefully you quickly identify very little that needs manual testing to free you up to explore the system in other ways.

Naturally, there will be some skeptics. So QA also needs to educate the team in the value of this approach, and why it’s so important that everyone view quality as a team effort. Just because we may help drive the discussion and define acceptance criteria, that doesn’t mean the tester owns the quality. Engineers may need to write more tests than they are used to, but it will only help their code as they build until every test passes. With everyone working together, engineers, QA, and designers all take part in defining quality, owning it, and delivering a better product.

Ashley Hunsberger is a Quality Architect at Blackboard, Inc. and co-founder of Quality Element. She’s passionate about making an impact in education and loves coaching team members in product and client-focused quality practices.  Most recently, she has focused on test strategy implementation and training, development process efficiencies, and preaching Test Driven Development to anyone that will listen.  In her downtime, she loves to travel, read, quilt, hike, and spend time with her family.

T: @aahunsberger
L: https://www.linkedin.com/in/ashleyhunsberger


 

How To Test Responsive Web Apps with Selenium

June 8th, 2015 by Dave Haeffner

The Problem

When testing a web application with a responsive layout you’ll want to verify that it renders the page correctly in the common resolutions your users use. But how do you do it?

Historically this type of verification has been done manually at the end of a development workflow — which tends to lead to delays and visual defects getting released into production.

A Solution

We can easily sidestep these concerns by automating responsive layout testing so we can get feedback fast. This can be done with a Selenium test, Applitools Eyes, and Sauce Labs.

Let’s dig in with an example.

An Example

NOTE: This example is built using Ruby and the RSpec testing framework. To play along, you’ll need Applitools Eyes and Sauce Labs accounts. They both have free trial accounts which you can sign up for here and here (no credit card required).

Let’s test the responsive layout for the login of a website (e.g., the one found on the-internet).

In RSpec, a test file is referred to as a “spec” and ends _spec.rb. So our test file will be login_spec.rb. We’ll start it by requiring our requisite libraries (e.g., selenium-webdriver to drive the browser and eyes_selenium to connect to Applitools Eyes) and specifying some initial configuration values with sensible defaults. (more…)

Test Automation KPIs

June 3rd, 2015 by Greg Sypolt

One of the interesting things about automation is that it frees you up from time-intensive manual testing, allowing you to spend time on strategic elements—because if you do not spend time on strategy, your capabilities as a team will not grow. And part of that growth means focusing on valuable metrics— metrics that will help you learn, and improve your processes.

Once you have processes in place, the next crucial step is to invest in automation. Automation helps you work faster, and makes your work consistent, traceable, and shareable, which is also imperative. All this comes only after establishing the right KPIs (key performance indicators).

Automation: Deliver Faster, from Months to Minutes

Ask yourself this question: without CI (continuous integration), how long would it take your organization to deploy a change that involves just one line of code? For instance, say your organization sets an objective to deploy a change in production within 30 minutes. To achieve this objective, everyone has to agree on the tools and processes that are needed for an easy button approach (aka continuous integration).

Let’s review the roles, team responsibilities, and the CI process. (more…)

Leverage Your QA Team Upfront

May 15th, 2015 by Ashley Hunsberger

All too often, QA is perceived as the bottleneck in getting software out the door.  This makes sense when you only see QA as just the “bug finder” in a world where you build, test, and release. But how can you leverage your QA team to improve your pipeline to delivery? There are many ways that you can do this, but let’s look at using defect prevention and analysis to test wisely and improve your pipeline.

Prevention

The scenario: a designer has sent specifications (and if you’re lucky, wireframes) for a new feature.  Depending on your development process, you would probably see the following happen:

  1. A developer writes some code
  2. A tester goes off and writes some tests
  3. Once the developer finishes code, the tester executes the tests, finds bugs, and tells the developer to fix the bugs that were found.

The above scenario seems simple, however at the later stages often one finds that the test cases do not 100% align with code. To fix this, what if you changed this process to look more like this?

(more…)