Posts Tagged ‘Q&A’

Should You Have a Dedicated Automation Team Within Your QA Department?

September 1st, 2015 by Israel Felix

If you’ve led or managed QA teams that have included an automation test team, you’ve probably been in a situation where you had to decide whether you should keep them on board. Normally the decision needs to be made when there is a change in leadership, wherein the new management comes with a mandate to consolidate groups and reduce costs. This situation also tends to arise when working with startups or small companies when they are ready to put together or augment their QA teams. So should you have a dedicated automation team?

Typically, there are two camps with regards to dedicated automation teams. There are those who believe that we should have dedicated automation teams, and those who believe that QA engineers should handle manual testing and automation testing. From my experience working in QA within both small and large companies, I almost always prefer to have a dedicated automation team. However, there are a few scenarios where having a QA team that takes on both roles might make sense.

Time to Market

For automation to be done right, it needs to be a full-time job. From developing the framework and creating the libraries and scripts for different platforms to executing and debugging failures — it will all simply consume too much of an engineer’s time and compromise the actual testing and release date. As you already know, time to market and keeping a release on schedule is top priority, so testing needs to get done, no matter what. (more…)

Guest Post: Functional Testing in 2016 – Forecast

April 15th, 2015 by Chris Riley

Massive changes in the development world are good and extreme for devs, but quality assurance (QA) teams are impacted as much, if not more. They may be taking on more tasks, looking at new tools, and thinking about new ways to execute their growing test suites. And looking forward, QA in the future looks much different than it does today. It is moving so fast that the changes – both good and bad – will be even more obvious by next year. Here is what QA looks like in 2016. (more…)

Recent Updates to Selenium Builder

December 11th, 2014 by Bill McGee

Great news for Enterprise folks and those who are newer to Selenium! Updates were made on Selenium Builder recently. (more…)

Sign Up for the First-Ever Appium Roadshow on August 20th in New York City

July 30th, 2014 by Bill McGee

appium_logoWe don’t know if you heard, but mobile is kind of a big deal.

Naturally, Appium – the only open source, cross-platform test automation tool for native, hybrid, and mobile web apps – emerged out of the need to Test All The (Mobile) Things.  Last May, battle-tested Appium 1.0 was released, and now this Appium show is hitting the road!

Details and ticket links below. Hope to see you in New York!


Sign Up for the First-Ever Appium Roadshow on August 20th

Appium Roadshow – NYC is a two part, day-long event held on Wednesday, August 20 at Projective Space – LES in Manhattan’s Lower East Side.

Part 1 – Appium in the Wild

8:30 AM – 1:00 PM – Free

The morning session will showcase presentations from Gilt Groupe, Sharecare, Softcyrlic, and Sauce Labs. Topics will cover real-world examples, lessons learned, and best practices in mobile app test automation using Appium. Featured speakers include:

  • Matthew Edwards – Mobile Automation Lead, Aquent
  • Daniel Gempesaw – Software Testing Architect, Sharecare
  • Matt Isaacs – Engineer, Gilt Groupe
  • Jonathan Lipps – Director of Ecosystem and Integrations, Sauce Labs
  • Sundar Sritharan – Delivery Manager, Softcrylic

This event is free. Breakfast and lunch included. Reserve your seat now – register here.

Part 2 – Appium Workshop

1:30 PM – 5:30 PM – $100

Matthew Edwards, a leading contributor to the Appium project, will provide a hands-on workshop to help you kick start your Appium tests.  He’ll talk you through how to set up the environment needed for native iOS and Android automation with Ruby.  You’ll then download and configure the to enable test writing. Then, Matthew will demonstrate how to kick up an Appium server and then run a test.

This event is limited to just 40 participants. Reserve your seat now – register here.


Guest Post: Automating Mobile Testing at Gilt – A Follow Up

June 19th, 2014 by Bill McGee

This post comes from our friends at Gilt, who are using Appium to automate their mobile testing.  Check out their first post here.

It’s been a while since we’ve posted in detail about our automation and testing process here on the Mobile Team at Gilt. Much has changed in the past few months, and the past couple of weeks in particular have been pretty exciting on the testing front. First off, we had the opportunity to give a talk about how we approach native mobile testing, to a packed house at the Brooklyn iOS Meetup. Second, Appium 1.0 – now already 1.1 – was officially released! Cue the champagne and fireworks! In seriousness though, the Appium milestone release is a pretty big deal for us.

Some History

We’ve written briefly about Appium before, and how it fits in with our overall process. About a year ago, Appium popped up on our radar here at Gilt after one of our engineers saw a talk about it at SeConf. We were intrigued and did a bit of experimenting, and after discovering that we’d be able to leverage some of our existing Selenium infrastructure, as well as our in house Selenium talent, we deciding to adopt Appium as our testing framework of choice.

Appium had been in development for some time by this point, but it was still early days for the project, and it had some rough edges. Tag names were inconsistent, and the XPath locator strategy was pretty sad. We’ll get into that later…

Appium’s pace of development was really fast, and things changed regularly. Often the changes would be drastic and would break our tests.
Stability also seemed to fluctuate. On some versions, things were terrific, and our tests would run smoothly. Other times, tests would fail or crash for no apparent reason. The failures weren’t always Appium’s fault, but they were still pretty frustrating. Once or twice we wondered why we were even bothering and considered switching back to KIF. We persisted though, and we’re really happy we did, because Appium has grown up!

So,  What’s All The Fuss About?

Appium now implements the defined part of the Mobile JSON Wire Protocol. This is basically an extension to the selenium wire protocol, that provides support for mobile specific actions. Desired capability keys seem to have been finalized by this spec, which is great, since these have changed a couple times over the past year, causing a couple of headaches. It’s also good to see some bigger companies getting involved here.

For us though, the toughest issue has been stability. Countless hairs have been turned grey due to tests failing or crashing for no apparent reason. While we’ve jumped through a variety of other hoops in our quest to solve the overall flakiness issue (page objects, judicious waiting), Appium has made huge strides in this area. This is no small achievement, given that Appium is really proxying commands to UIAutomation, which itself is pretty fragile and prone to random crashing.

We’re also super happy to see that force-quit-instruments no longer shuts test sessions down during long waits or element searches.
Most surprisingly, page source is now XML based. XML? It’s like it’s 1999 all over again! Actually this change has some pretty amazing benefits. The biggest of which is that we can now perform “real” XPath queries on our page source, and Appium is now using a compliant XPath parser. Hooray! We complained before about Appium’s XPath engine being pretty lousy. In olden times, the XPath engine was regex based and would search for the desired element(s) in a JSON representation of the page source. Dreadful stuff… Needless to say, this was inefficient and only worked correctly with “just right” XPath queries. This basically left Appium users with only two legitimate locator strategies. We’re super happy to see that this has been resolved.

Too Good To Be True?

We’re generally really happy with the way Appium works, but there are a handful of issues that still cause some pain for us. Error reporting is not Appium’s strong suit. While overall reliability is orders of magnitude better in the latest release, it’s still incredibly hard to identify the exact problem when failures occur. Integrating Appium with your Xcode workflow is left up to you, and while this isn’t terribly hard, it would be nice to see tighter Xcode integration. I still stand by our previous advice of steering clear of any non-standard UI tricks and hacks. Minimize the use of overlays, and do whatever you can to avoid passthrough views. If anything, Appium will push you to strive for UI simplicity and to get serious about accessibility. The proposed touch gestures API might provide some flexibility here, but we’ll have to wait and see.

Coming Soon

We’re still busy getting to grips with the changes brought by this milestone release. The increased interest in the the Mobile JSON Wire Protocol by some bigger companies has pushed our confidence in Appium up a couple notches. While the spec is not yet complete, it does propose some really exciting features down the line. Notably, we’re really excited by the prospect of server-side waits. The way we manage waiting for elements has had a huge impact on the reliability and speed of our tests, and so it’s really exciting to see this part of the process getting some love.

The Bottom Line

Almost all reliability issues that plagued us in earlier versions appear to have been resolved. We’re now able to spend more time working on tests, and much less time wrestling with Appium and Instruments. Prior to this release, our tests were totally untrustworthy. The changes made leading up to the 1.0 release, combined with better test practices on our part, have dramatically boosted our confidence in our automated tests. Test results are now taken seriously and are becoming an invaluable aid when performing QA work on our apps. We’re really excited to see how the Mobile JSON Wire Protocol evolves, and what comes next for Appium.

By Matt Isaccs of Gilt Groupe for Sauce Labs

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.


Guest Post: Bridging the Test Divide – Beyond Testing For A Release

June 16th, 2014 by Bill McGee

This is the second of a three part series by Matthew Heusser, software delivery consultant and writer. 

When I start to think about testing, I think about it in two broad strokes: new feature testing and release-testing. New feature testing tries to find problems with something new and specific, while release-testing happens after “code complete”, to make sure the whole system works together, that a change here didn’t break something there.

Release-testing (which some call regression testing) slows down the pace of release and delays feedback from our customer. Release-testing also increases cycle time – the time from when we begin work on a feature until it hits production. Over time, as our software become more complex, the amount of testing we want to do during release testing goes up.

Meanwhile, teams want to ship more often, to tighten the feedback loop.

Today I am going to talk about making release testing go away – or at least drastically reducing it.

It all starts during that tutorial in Spain I wrote about last time.

Two Worlds

The frequency of release for the people in my tutorial was very diverse, but two groups really struck me — the telecom that had a four-month test-release cycle, and the Latvian software team with the capability to deploy to production every single day.

That means arriving at the office the morning, looking at automated test runs, and making a decision to deploy.

There is a ‘formula’ to make this possible. It sounds simple and easy:

  • Automate a large number of checks on every build
  • Automate deploy to production
  • Continuously monitor traffic and logs for errors
  • Build the capability to rollback on failure

That transforms the role of test from doing the “testing we always do” to looking at the risk for a given release, lining it up against several different test strategies, and balancing risk, opportunity, reward, and time invested in release-testing?

The trick is to stop looking at the software as a big box, but instead to see it as a set of components. The classic set of components are large pieces of infrastructure (the configuration of the web server, the connections to the database, search, login, payment) and the things that sit on top of that – product reviews, comments, static html pages, and so on. Develop at least two de-ploy strategies — one for audited and mission-critical systems (essential infrastructure, etc) and another for components and add-ons.

We’ve been doing it for years in large IT organizations, where different systems have different release cycles; the trick is to split up existing systems, so you can recognize and make low-risk changes easier.

This isn’t something I dreamed up; both Zappos and Etsy have to pass PCI audits for financial services, while Zappos is part of Amazon and publicly traded. Both of these organizations have a sophisticated test-deploy process for parts of the application that touch money, and a simpler process for lower-risk changes.

So split off the system into different components that can be tested in isolation. Review the changes (perhaps down to the code level) to consider the impact of the change, and test the appropriate amount.

This can free up developers to make many tiny changes per day as long as those changes are low risk. Bigger changes along a theme can be batched together to save testing time — and might mean we can deploy with still considerably less testing than a ‘full’ site retest.

But How Do We Test It?

A few years ago, the ideal vision of getting away from manual, documented test cases was a single ‘test it’ button combined with a thumbs up or down at the end of an “automated test run.”

If the risk is different for each release, and we are uncomfortable with our automation, then we actually want to run different tests for each release — exactly what thinking testers (indeed, anyone on the team) can do with exploratory testing.

So let the computers provide some automated checks, all the time. Each morning, maybe every half an hour, we get a report, look at the changes, and decide what is the right thing for this re-lease. That might mean full-time exploratory testing of major features for a day or two, it might be emailing the team and asking everyone to spend a half hour testing in production.

This result is grown up software testing, varying the test approach to balance risk with cost.

The first step that I talked about today is separating components and developing a strategy that changes the test effort based on which parts were changed. If the risk is minimal, then deploy it every day. Hey, deploy it every hour.

This formula is not magic. Companies that try it find engineering challenges. The first build/deploy system they write tends to become hard to maintain over time. Done wrong continuous testing creates systematic and organizational risk.

It’s also a hard sell. So let’s talk about ways to change the system to shrink the release-test cycle, deploy more often, and reduce risk. The small improvements we make will stand on their own, not threaten anyway — and allow us to stop at any time and declare victory!

A Component Strategy

that_badWhen a company like says that new programmers commit and push code to production the first day, do they really mean modifications to payment processing, search, or display for all products?

Of course not.

Instead, programmers follow a well-written set of directions to … wait for it … add the new user to the static HTML ‘about us’ page that lists all the employees, along with an image. If this change generates a bug, that will probably result in an X over an image the new hire forgot to upload, or maybe, at worst, break a div tag so the page mis-renders.

A bad commit on day one looks like this – not a bungled financial transaction in production.

How much testing should we have for that? Should we retest the whole site?

Let’s say we design the push to production so the ‘push’ only copies HTML and image files to the webserver. The server is never ‘down’, and serves complete pages. After the switch, the new page appears. Do we really need to give it the full monty, the week-long burn down of all that is good and right in testing? Couldn’t the developer try it on a local machine, push to stag-ing, try again, and “just push it?”

Questions on how?

More to come.

By Matthew Heusser – for Sauce Labs

Stay tuned next week for the third part of this mini series! You can follow Matt on Twitter at @mheusser.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Guest Post: Test Lessons at ExpoQA

June 6th, 2014 by Bill McGee

This is the first of a three part series by Matthew Heusser, software delivery consultant and writer. 

Every now and again and opportunity comes along that you just can’t refuse. Mine was to teach the one-day version of my class, lean software testing, in Madrid, Spain, then again the following week in Estonia. Instead of coming back to the United States, I’ll be staying in Europe, with a few days in Scotland and a TestRetreat in the Netherlands.

And a lot of time on airplanes.

The folks at Sauce Labs thought I might like to take notes and type a little on the plane, to share my stories with you.

The first major hit in Madrid is the culture shock; this was my first conference where English was not the primary language. The sessions were split between English and Spanish, with translators in a booth making sure all talks were available in all languages.

The Testing Divide

Right now, in testing, I am interested in two major categories: The day to day work of testing new features and also the work of release-testing after code complete. I call this release testing a ‘cadence’, and, across the board, I see companies trying to compress the cadence.

My second major surprise in Madrid is how wide the gap is —and I believe it is getting wider —between legacy teams that have not modernized and teams starting from scratch today. One tester reported a four-month cycle for testing. Another team, relying heavily on Cucumber and Selenium, were able to release every day.

Of course, things weren’t that simple. The Lithuanian team used a variety of techniques I can talk about in another post to reduce risk, something like devOps, which I can talk about in another post. The point here is the divide between the two worlds.

Large cadences slow down delivery. They slow it down a lot; think of the difference between machine farming in the early 20th century and the plow and horse of the 19th.

In farming, the Amish managed to survive by maintaining a simple life, with no cars, car insurance, gasoline, or even electricity to pay for. In software, organizations that have a decades-long head start: banks, insurance companies, and pension funds, may be able to survive without modernization.

I just can’t imagine it will be much fun.

Batches, Queues and Throughput

Like many other conferences, the first day of ExpoQA is tutorial day, and I taught the one-day version of my course on lean software testing. I expected to learn a little about course delivery, but not a lot —so the learning hit me like a ton a bricks.

The course covers the seven wastes of ‘lean’, along with methods to improve the flow of the team – for example, decreasing the size of the measured work, or ‘batch size’. Agile software development gets us this for free, moving from ‘projects’ to sprints, and within sprints, stories.

In the early afternoon we use dice and cards to simulate a software team that has equally weighted capacity between analysis, dev, test and operations —but high variability in work size. This slows down delivery. The fix is to reduce the variation, but it is not part of the project, so what the teams tend to do is to build up queues of work, so any role never runs out of work.

What this actually does is run up the work in progress inventory – the amount of work sitting around, waiting to be done. In the simulation I don’t penalize teams for this, but on real software projects, ‘holding’ work created multitasking, handoffs, and restarts, all of which slow down delivery.

My lesson: Things that are invisible look free —and my simulation is far from perfect.

After my tutorial it is time for a conference day – kicked off by Dr. Stuart Reid, presenting on the new ISO standard for software testing. Looking at the schedule, I see a familiar name; Mais Tawfik, who I met at WOPR20.Mais is an independent performance consultant; today she is presenting on “shades of performance testing.”

Performance Test Types

Starting with the idea that performance testing has three main measurements: Speed, Scalability, and Stability, Mais explains that there are different types of performance tests, from front-end performance (javascript, waterfalls of HTTP requests, page loading and rendering) to back-end (database, webserver), and also synthetic monitoring – creating known-value transactions continuously in production to see how long they take. She also talks about application usage patterns – how testing is tailored to the type of user, and how each new release might have new and different risks based on changes introduced. That means you might tailor the performance testing to the release.

At the end of her talk, Mais lists several scenarios and asks the audience what type of performance test would blend efficiency and effectiveness. For example, if a release is entirely database changes, and time is constrained, you might not execute your full performance testing suite/scripts, but instead focus on rerunning and timing the database performance. If the focus on changes is the front end, you might focus on how long it takes the user interface to load and display.

When Mais asks if people in the organization do performance testing or manage it, only a handful of people raise their hands. When she asks who has heard of FireBug, even less raise their hand.

Which makes me wonder if the audience is only doing functional testing. If they are, who does the performance testing? And do they not automate, or do they all use Internet Explorer?

The talk is translated; it is possible that more people know these tools, it was just that the translator was ‘behind’ and they did not know to raise their hands in time.

Here’s hoping!

Time For A Panel

At the end of the day I am invited to sit on a panel to discuss the present (and future) of testing, with Dr. Reid, Dorothy Graham, Derk-Jan De Grood, Celestina Bianco and Delores Ornia. The questions include, in no particular order:

  •             Will testers have to learn to code?
  •             How do we convince management of the important of QA and get included in projects?
  •             What is the future of testing? Will testers be out of a job?
  •             What can we do about the dearth of testing education in the world today?

For the problem with the lack of education, Dorothy Graham points to Dr. Reid and his standards effort as a possible input for university education.

When it is my turn, I bring up ISTQB The International Software Testing Qualifications Board. – if ISTQB is so successful (“300,000 testers can’t be wrong?”) then why is the last question relevant? Stefaan Luckermans, the moderator, replied that with 2.9 Million testers in the world, the certification had only reached 10%, and that’s fair, I suppose. Still, I’m not excited about the quality of testers that ISTQB turns out.

The thing I did not get to say, because of time, that I want to do is point out that ISTQB is, after all, just a response to a market demand for a 2-3 day training certification. What can a trainer really do in 2-3 days? At most, maybe, teach a single technical tool, turn the lightbulb of thinking on, or define a few terms. ISTQB defines a few terms, and it takes a few days.

The pursuit of excellent testing?

That’s the game of a lifetime.

By Matthew Heusser – for Sauce Labs

Stay tuned next week for part two of this mini series! You can follow Matt on Twitter at @mheusser.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Open Source Stories: Q&A with Lo-Dash

March 28th, 2014 by Bill McGee

LOAs you may already know, we LOVE open source. That’s why we created Open Sauce; a way to give open source projects the ability to test their projects for free on our cloud.

In the span of a year, we now have over 800 OSS projects testing on Sauce! To celebrate, we’re exploring some of the different projects tested on Sauce with a mini series of Q&A’s with users who have produced notable projects using Open Sauce.

Today we’re featuring an interview with John-David Dalton of Lo-Dash. Get the scoop below.

Can’t get enough of John-David?  Neither can we. You can hear him speak at Front End Ops Conf 2014. Register and get tickets now!

What is your OSS project, and what is its goal?
Lo-Dash is a utility library with an emphasis on consistency, customization, & performance.

What kind of testing does the project do, both on and off Sauce?
On Sauce we automate testing of Lo-Dash in legacy, compat, mobile, modern, & underscore builds (non-minified/minified and modularized AMD builds of each) against Lo-Dash/Underscore/Backbone unit tests. We test against IE6-11, old and new Firefox, Chrome, and Opera. We even test IE compat modes. Off Sauce we automate testing in Node, Narwhal, Phantomjs, Rhino, Rhino -require, & Ringo.

How has testing with Open Sauce helped the project?
Testing used to consist of ~2 days of manual testing. With Sauce & its paralleled jobs testing is automated to ~30 minutes.

Do you have any advice regarding testing for other open source projects?
Do it. Start small and build then pretty soon you’ll have a nice collection of tests. They’ll save you so much grief and build trust with your users/developers.

What’s next for the project?
We are building up to our v3.0 release with lots of features & some changes. Check out our roadmap

How can people learn more about the project or get involved?
They can go to or

How have you used Open Sauce? Leave us a comment below, Tweet at us, or submit your story by way of a guest post!