Posts Tagged ‘Continuous Testing’

Getting the Existing Team On Board with Automation (Scripts)

August 27th, 2015 by Greg Sypolt

Introduction

In an attempt to do more with less, organizations want to test their software adequately, as quickly as possible. Businesses are demanding quick turnaround, pushing new features and bug fixes to production within days, along with quality. Everyone knows manual testing is labor-intensive and error-prone, and does not support the same kind of quality checks that are possible through the use of an automated test. Nobody likes change, but it’s time to educate your team on the importance of onboard automated testing.

The only way to make sense out of change is to plunge into it, move with it, and join the dance.  – Alan Watts

Everyone has had a job interview at some point in their lives, right? It is important to be prepared! The first few minutes of an interview are a make or break moment. Why? Because first impressions can have long-lasting effects. Never underestimate the power of first impressions. The same principle applies when onboarding automation to an existing manual testing team. Your initial presentation to your team or organization should be treated like a job interview. Be prepared. Deliver expectations and explain responsibilities — it’s critical since it is normal for employees to have an emotional reaction to anything they view as a job threat.

Why automated testing?

If things are going well, why do we want to implement automated tests? The demand is to do more with less, which makes manual testing an impossible task, but introducing automated testing into an existing software development lifecycle can also be daunting. However, when implemented, automated testing is a valuable asset that shortens testing cycles and helps teams become more agile.

marketingautomation1011

image source: http://marketing-works.net/

(more…)

Not Just Faster Releases; Better Understanding

May 7th, 2015 by Chris Riley

This is a guest post by Chris Riley, a technologist who has spent 12 years helping organizations transition from traditional development practices to a modern set of culture, processes and tooling.

In DevOps there are several processes and tools that go beyond their face value. System Analytics, Application Performance Monitoring, and Continuous Integration (CI) go far beyond their core abilities. In particular, CI not only changes the speed and quality of releasing code, it improves communication and finds bugs in the software development process itself.

One of my biggest messages for companies moving to modern software delivery is the idea of being deliberate. Being deliberate means picking a culture, process, and tools that focus on results and how to attain them. Too often tools, especially those that are open source, are adopted on a whim without much forethought. This is the converse of being deliberate; allowing the tools to define the process and culture for you.

When organizations follow the ‘deliberate’ approach, they naturally get to a place where they move faster and what they have built is sustainable. A huge component of getting to this place is CI. No DevOps shop can survive without a continuous integration process. It allows front-end developers to participate in quality, find bugs before QA, test new functionality faster than ever. (more…)

Continuous Testing in Practice [WEBINAR]

February 18th, 2015 by Bill McGee

As web and mobile application software increases in complexity, the number and frequency of tests grows exponentially. But managing your tests with sub-optimal continuous integration (CI) workflows can lead to bottlenecks, delays, and lost developer productivity.

In our next webinar, Continuous Testing in Practice, Ophir Prusak from BlazeMeter and Abhijit Pendyal from Sauce Labs will show you how to integrate automated testing into your CI process so that you can test early and often to speed up deployment.

Ophir and Abhijit will cover:

  • Why Continuous Testing is so important today
  • How to ensure testing keeps pace with agile development cycles
  • The end-to-end flow of a continuous testing process
  • How to implement continuous automated functional & performance testing
  • How to integrate continuous testing with your existing tools

Join us for this presentation on Tuesday, February 24th at 10am PST/1pm EST. There will be a Q&A with both Ophir and Abhijit following the end of the presentation.

Click HERE to register today.

Automated Testing News: Link Round-Up

January 23rd, 2015 by Bill McGee
Happy Friday! Here’s a quick round-up of some pieces on automated testing, why functional and performance testing should be done simultaneously, the case for Continuous Testing in the Cloud,  how to take your career to the next level, QTP/UFT VS Selenium, and what the top 10 tools were this week on Stackshare.  See below for snippets and links to the full articles.

(more…)

Recap: Continuous Testing in the Cloud [WEBINAR]

June 30th, 2014 by Bill McGee

Last week our own Michael Sage discussed the topic of  Continuous Testing in the Cloud for our latest webinar.

In it, he went over how to take advantage of cloud-hosted development resources to help increase release time and improve application quality as a best practice. He also talked about how to use Sauce Labs to securely execute your Selenium tests in parallel and reduce the time it takes to run your critical integration and acceptance tests.

Missed the webinar? You can listen to the recording here, or check out the slides below.

Guest Post: Bridging the Test Divide – Beyond Testing For A Release

June 16th, 2014 by Bill McGee

This is the second of a three part series by Matthew Heusser, software delivery consultant and writer. 

When I start to think about testing, I think about it in two broad strokes: new feature testing and release-testing. New feature testing tries to find problems with something new and specific, while release-testing happens after “code complete”, to make sure the whole system works together, that a change here didn’t break something there.

Release-testing (which some call regression testing) slows down the pace of release and delays feedback from our customer. Release-testing also increases cycle time – the time from when we begin work on a feature until it hits production. Over time, as our software become more complex, the amount of testing we want to do during release testing goes up.

Meanwhile, teams want to ship more often, to tighten the feedback loop.

Today I am going to talk about making release testing go away – or at least drastically reducing it.

It all starts during that tutorial in Spain I wrote about last time.

Two Worlds

The frequency of release for the people in my tutorial was very diverse, but two groups really struck me — the telecom that had a four-month test-release cycle, and the Latvian software team with the capability to deploy to production every single day.

That means arriving at the office the morning, looking at automated test runs, and making a decision to deploy.

There is a ‘formula’ to make this possible. It sounds simple and easy:

  • Automate a large number of checks on every build
  • Automate deploy to production
  • Continuously monitor traffic and logs for errors
  • Build the capability to rollback on failure

That transforms the role of test from doing the “testing we always do” to looking at the risk for a given release, lining it up against several different test strategies, and balancing risk, opportunity, reward, and time invested in release-testing?

The trick is to stop looking at the software as a big box, but instead to see it as a set of components. The classic set of components are large pieces of infrastructure (the configuration of the web server, the connections to the database, search, login, payment) and the things that sit on top of that – product reviews, comments, static html pages, and so on. Develop at least two de-ploy strategies — one for audited and mission-critical systems (essential infrastructure, etc) and another for components and add-ons.

We’ve been doing it for years in large IT organizations, where different systems have different release cycles; the trick is to split up existing systems, so you can recognize and make low-risk changes easier.

This isn’t something I dreamed up; both Zappos and Etsy have to pass PCI audits for financial services, while Zappos is part of Amazon and publicly traded. Both of these organizations have a sophisticated test-deploy process for parts of the application that touch money, and a simpler process for lower-risk changes.

So split off the system into different components that can be tested in isolation. Review the changes (perhaps down to the code level) to consider the impact of the change, and test the appropriate amount.

This can free up developers to make many tiny changes per day as long as those changes are low risk. Bigger changes along a theme can be batched together to save testing time — and might mean we can deploy with still considerably less testing than a ‘full’ site retest.

But How Do We Test It?

A few years ago, the ideal vision of getting away from manual, documented test cases was a single ‘test it’ button combined with a thumbs up or down at the end of an “automated test run.”

If the risk is different for each release, and we are uncomfortable with our automation, then we actually want to run different tests for each release — exactly what thinking testers (indeed, anyone on the team) can do with exploratory testing.

So let the computers provide some automated checks, all the time. Each morning, maybe every half an hour, we get a report, look at the changes, and decide what is the right thing for this re-lease. That might mean full-time exploratory testing of major features for a day or two, it might be emailing the team and asking everyone to spend a half hour testing in production.

This result is grown up software testing, varying the test approach to balance risk with cost.

The first step that I talked about today is separating components and developing a strategy that changes the test effort based on which parts were changed. If the risk is minimal, then deploy it every day. Hey, deploy it every hour.

This formula is not magic. Companies that try it find engineering challenges. The first build/deploy system they write tends to become hard to maintain over time. Done wrong continuous testing creates systematic and organizational risk.

It’s also a hard sell. So let’s talk about ways to change the system to shrink the release-test cycle, deploy more often, and reduce risk. The small improvements we make will stand on their own, not threaten anyway — and allow us to stop at any time and declare victory!

A Component Strategy

that_badWhen a company like etsy.com says that new programmers commit and push code to production the first day, do they really mean modifications to payment processing, search, or display for all products?

Of course not.

Instead, programmers follow a well-written set of directions to … wait for it … add the new user to the static HTML ‘about us’ page that lists all the employees, along with an image. If this change generates a bug, that will probably result in an X over an image the new hire forgot to upload, or maybe, at worst, break a div tag so the page mis-renders.

A bad commit on day one looks like this – not a bungled financial transaction in production.

How much testing should we have for that? Should we retest the whole site?

Let’s say we design the push to production so the ‘push’ only copies HTML and image files to the webserver. The server is never ‘down’, and serves complete pages. After the switch, the new page appears. Do we really need to give it the full monty, the week-long burn down of all that is good and right in testing? Couldn’t the developer try it on a local machine, push to stag-ing, try again, and “just push it?”

Questions on how?

More to come.

By Matthew Heusser – matt.heusser@gmail.com for Sauce Labs

Stay tuned next week for the third part of this mini series! You can follow Matt on Twitter at @mheusser.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.