Testing and Continuous Integration: Making Everything Work Together

May 17th, 2016 by Hemant Jain

Continuous integration (CI) has emerged as one of the most efficient ways to develop code. But testing has not always been a major part of the CI conversation.

In some respects, that’s not surprising. Traditionally, CI has been all about speeding up the coding, building, and release process. Instead of having each programmer write code separately, integrate it manually, and then wait until the next daily or weekly build to see if the changes broke anything, CI lets developers code and compile on a virtually continuous basis. It also means developers and admins can work together seamlessly, since the programming and build processes are always in sync.

Unfortunately, the quality assurance team does not necessarily reap the same benefits. While CI assures that your app keeps building successfully as the code is continually updated, it doesn’t automatically test how new builds behave within different types of environments. An otherwise well-run CI operation might require app testing to be done separately, on a non-continuous basis, instead of building it into the rest of the process.

This poses real problems for an organization. Unless you add automated testing to your CI mix, you could end up with an app that users can download and install properly, but which suffers from critical usability issues in certain browsers or operating systems. Arguably, an app that installs successfully but frustrates users due to lack of testing is worse than one that doesn’t install at all. (more…)

JIRA is Just a Click Away with Our New Plugin

May 3rd, 2016 by Ken Drachnik

As more and more devs work in agile teams, they need tools to plan, track, and release software and many of them use JIRA. As a tracking tool, JIRA is amazing for collaboration and project planning. For many teams, JIRA is the place of record for everything in the software development lifecycle. We have found that many of our customers use JIRA and the #1 product ask was to integrate Sauce Labs’ test results with JIRA so it would be simple to track all the tests associated with a project in one place.

Let’s say you are running an automated or manual test on Sauce Labs and find a bug. You want to add it to JIRA so that someone on your team can take a look or so that it can be prioritized in the backlog. Historically, one would have to download all the Sauce assets, login to JIRA, create a ticket, and upload the assets again. This can be tedious when you’re running lots of tests.

With Sauce Labs for JIRA, this is all simplified and automated. With the click of a button you can now create a JIRA ticket directly from your Test Details page. The plugin gives you the option to upload the screenshots, logs, and video link to your tests, making it easy to share out among your team.

To download Sauce Labs for JIRA, visit the Atlassian Marketplace: https://marketplace.atlassian.com/plugins/sauce-jira-integration/cloud/overview.  To read more, visit our JIRA integration Docs page.

Happy Testing!

The Sauce Labs Ecosystems & Integrations Team

Recap: Easy Continuous Deployment You Can Trust (Webinar)

April 14th, 2016 by Bill McGee

Thanks to everyone who joined us for our recent webinar, “Easy Continuous Deployment You Can Trust”, featuring Solano Labs Founding Engineer, Brian Kendzior, and Sauce Labs Solutions Architect, Neil Manvar.

In their presentation, Brian and Neil demonstrated a continuous deployment release process that used Solano CI, Sauce Labs, and AWS CodePipeline. The release process included smoke, unit, integration and browser tests to guarantee issue-free deployments.

Brian and Neil also demonstrated how to:

  • Build your software release pipeline by connecting different steps into an automated workflow using AWS CodePipeline
  • Run your automated web tests on any browser and operating system combination using Sauce Labs
  • Auto-parallelize your test runs with a Continuous Integration server using Solano CI

Want to learn more about automated testing and Continuous Integration? Download our free report, “How to Get the Most out of Your CI/CD Workflow Using Automated Testing”. 

Access the recording HERE and view the slides below: 

Test Faster and Smarter by Testing in Production

April 14th, 2016 by Chris Riley

You may dread the term testing in production (TiP). The thought of potential loss of data, downtime, and a damaged reputation to organizations can be daunting. But things need not be that way. In fact, today, testing in production is used by some of the biggest organizations with much success. But can it become a reality for your team?

Accident or Intentional?

Testing in production is not a completely new concept. In fact, you’ve probably seen it in action more often than you imagine. Think of an app that you’ve released or one that you know of that was poorly tested. You likely spent the next few weeks firefighting, and got it to be functional faster than you thought possible. In this case, you were forced to test in production. But, what if you could hone the art of testing in production and use it to your advantage? What if you could spot and fix issues so much faster than you do today? What if you could influence development from start to end? What if you could do all this without risking the reputation of your team or organization? That’s the promise of TiP, and it’s worth a second glance. (more…)

Recap: Achieve True Continuous Integration with Sauce Labs and Microsoft Visual Studio Team Services (Webinar)

March 11th, 2016 by Bill McGee

Thanks to everyone who joined us for our recent webinar, “Achieve True Continuous Integration with Sauce Labs and Microsoft Visual Studio Team Services”, featuring Sauce Labs Product Manager Jack Moxon. In his presentation. Jack shows the new Sauce Labs plugin for Visual Studio Team Services (VSTS) and how to launch tests on Sauce Labs as part of their VSTS build, enabling Continuous Integration (CI).

Jack also shows how the plugin allows users to launch Sauce Connect – a tunnel that allows customers to securely test pre-production web and mobile apps in the cloud. The plugin also integrates Sauce Labs test results and detailed reports (like videos, screenshots and logs) back into Visual Studio and TFS to enable collaboration and expedite the debugging process.

The webinar covers:

  • How Sauce Labs and Visual Studio Team Services (VSTS) are integrated for Continuous Integration
  • Launching automated tests as a part of a VSTS build
  • How to test pre-production apps securely behind your firewall
  • Sharing detailed test results and reports to collaborate and debug faster

Want to learn more about the Sauce Labs plugin for VSTS? We’ll be exhibiting at Microsoft Build in San Francisco March 30 – April 1, 2016. The conference is sold out but you can add a reminder to your calendar to make sure you don’t miss the live stream.

Access the recording HERE and view the slides below: 

Recap: Continuous Testing Meets the Classroom: Testing Online, Interactive Curriculum at Code.org [Webinar]

February 12th, 2016 by Bill McGee

Thanks to everyone who joined us for our recent webinar, “Continuous Testing Meets the Classroom: Testing Online, Interactive Curriculum at Code.org”, featuring Brian Jordan from Code.org. In his presentation, Brian discussed how Code.org approaches testing throughout the product development cycle, given their unique testing challenges – developing interactive, game-like curriculum for just the types of browsers you’d expect to find in school computer labs – from Internet Explorer 9 to iPads across 40+ languages.

The presentation also showed:

  • What Code.org’s open source automated testing stack looks like
  • What visual testing with Applitools, cross-browser Selenium tests on Sauce Labs, and live-site monitoring with New Relic and Honeybadger look like in practice
  • About the Code.org “Bug Collection” —real live examples of bugs detected before they hit production

Missed the presentation, want to hear it again, or share with a colleague?

Access the recording HERE and view the slides below.

Want to learn more about automated testing and Continuous Integration? Download our free report, “How to Get the Most out of Your CI/CD Workflow Using Automated Testing”.

Balancing Documentation With Speed

December 23rd, 2015 by Ashley Hunsberger

I’m sitting here thinking about my career over the last decade or so, thinking about how things once were, and how things have changed. One thing stands out to me: how much faster things are now. How fast we build, how fast we release, how quickly we have to test, how quickly we have to write. How do we balance documentation (which, let’s face it, could take a lot of time) with the need to get things out the door? How do we keep up in the continuous pipeline? It comes down to having a centralized strategy that consists of best practices, how to get through your day-to-day work, and working as a team to know what needs to be tested.

One Strategy to Rule Them All

One thing I’ve found that has helped to save time is to have a centralized space to reference when nothing really changes in HOW things need to be tested. These are things that do not change across projects or releases and should be considered as long-term and used as a reference. We used to do this for each epic and it drove me nuts — it ultimately led to tons of strategies with duplicate information throughout. Now, we have a reference wiki space and go over strategies that everyone (not just QA) should be familiar with. These strategies should also be a part of passing acceptance criteria, and your engineers need to know these strategies as well. (more…)

Merging Business Process With Continuous Delivery

November 20th, 2015 by Ashley Hunsberger

One of the biggest hurdles in getting to Continuous Delivery and truly being Agile does not lie within the development team itself. Change requires a mindset that all people (managers and executives too) must adopt. Just as it takes a village to raise a child, it also takes a (corporate) village to raise a new culture. The movement away from Waterfall to Agile and Continuous cannot be handled just by one person.

Does the following situation sound familiar? Someone asks you for an estimate for the Level of Effort (LOE) of a feature. Not just a t-shirt size, but in days — something to be delivered sometime that year. You don’t have much time to really dive into it (it could even be during the same meeting that you first hear about the new feature), but you have to know all the details up front. So you give a number that you aren’t supposed to be held to because of all sorts of caveats you listed. But, someone remembers that number. And since someone said you are Agile (and Continuous Delivery), now you have to not only meet that estimate you gave up front, you are now tied to an Agile sprint with no room for error. This goes against everything you say you are!

(more…)

Cattle, Not Pets – Use Automation Software To Provision Your Servers

October 6th, 2015 by The Sauce Labs Team

This guest post was written by Julian Dunn, Product Manager at Chef Software (@julian_dunn)

In Greg Sypolt’s earlier post on immutable infrastructure, he outlined practices for treating your servers more like cattle and not pets. It’s better to rebuild things if you can rather than spending calories on manually debugging and fixing long-lived servers. (Of course, you will inevitably have a few pets in your infrastructure; for instance, database servers that actually hold customer data that would be inconvenient to constantly rebuild.)

Automation software like Chef can help with the provisioning and automated setup of new cattle in a consistent way without having to resort to shell scripting or other ugly, hard-to-understand tools. Chef can also help you with maintaining consistency among the few pet servers that you do have.

One trend we’ve seen in the pets-to-cattle migration is the externalization of interface testing tools from long-lived, always-on test driver clusters to ephemeral instances or containers to drive UI tests, record the results, report them to a central dashboard and then destroy themselves. These cloud-native testing systems like Sauce Labs also provide a far richer set of functionality, like video recording and playback of the UI test, than can be achieved with artisanally-crafted, on-premise solutions. In other words, why mess around with having to configure and maintain Xvfb, Selenium, Webdriver and so on, yourself, for a boatload of platforms and platform versions when you can just use Sauce Labs?

(more…)

Continuous Integration – The Heart of DevOps

August 26th, 2015 by Chris Riley

Missed our earlier webinar, “Beyond the Release – CI That Transforms“? Check out the recap below.

In this webinar, we discussed the power of CI and possible considerations for the future. One of the more interesting aspects of the webinar was seeing the modern pipeline with the new Sauce CI Dashboard alongside Cloudbees’ automation templates. We also conducted a small CI survey at the beginning of the webinar, and ended with a Q&A.

The Power of CI

Continuous Delivery and Deployment (CD) steal the show in DevOps conversations, but the reality is that Delivery and Deployment are not for every organization, nor are they already widely adopted. In order to move on to delivery and deployment, organizations must get Continuous Integration (CI) right — unless they were built from day one with the DevOps framework, and did not have to fit the processes into existing environments.

The reason CI is so powerful is that it allows you to dip your toe into the modern delivery pipeline without the risk or complexity of building out delivery and deployment all at once – and with the potential of failure. You can consider CI as the on-boarding for DevOps. And from a process and tool standpoint, CI is nearly identical to delivery and deployment, it means that once you get it right you can easily move on.

Webinar – CI Survey Results

I’m pleased to say the results of the Sauce Labs CI survey were almost exactly what I expected, served with a side of surprise. For me, the most interesting aspect of the survey results is how they appear to be in conflict with the perceived high CI adoption and success rates already existing in the market. Let’s look at the results among 500+ attendees:

What Types of Automated Tests do you run?

  • Unit 28%
  • Functional 40%
  • Integration 27%
  • None 6%

6% of the attendees are not running automated tests at all! This was astonishing to me. I expected 1% at most, especially given the audience, because they are already familiar with automation. At a minimum, I would expect all companies to automate Unit testing. However there is a high likelihood that what this 6% is saying is that they have automated tests, just initiated manually.

The results also showed that many are running functional tests. This is great! However, only 27% are running integration tests. This is troubling because compared to the reported 45% who state they are doing CI already, the lack of integration testing would seem to contradict that statement. I suspect that this is a definition problem, where some may define CI as being simply a shared testing environment, and not really the CI process as described in the webinar. (more…)