A Getting Started Guide to Setting Up Jenkins

September 7th, 2016 by Greg Sypolt

The goal of this getting started guide is to help teams get Jenkins continuous integration (CI) servers configured, and discover how to make a newly deployed CI infrastructure fully operational. Jenkins is a leading open source CI server. It is flexible, providing hundreds of plugins to support building, testing, and deployment, and is capable of automating any project. Jenkins CI infrastructure can be deployed to on-prem, in the cloud using configuration management tools, and third-party vendor. For the purpose of this article, let’s assume that our Jenkins CI servers are deployed in the cloud and focus on configuration of Jenkins’ web interface. I will walk through various processes and steps to set up a new Jenkins CI server for production.

Recommended best practices for CI architecture

Don’t jump head first into configuration and creating a pipeline without planning, designing and establishing standards for your CI architecture. Taking the time to think about infrastructure first will enable a stable and restorable infrastructure. Let’s review a few recommended best practices to consider in your future CI pipeline.

Backup CI Server (Fail Safe)

It may seem obvious. Recommend setting up a backup process for Jenkins configuration. Script a Jenkins job to use the thinBackup plugin or S3 plugin to send the Jenkins configuration to an Amazon S3 (cloud storage).

Pipeline Configuration

Here are recommendations to consider: set environment variables (i.e., hidden passwords, SSH keys, API keys, etc.); security—create generic reusable jobs, naming conventions (i.e., jobs and environment variables); keep jobs small—modulization, scalable infrastructure that allows for auto-scaling of Slave nodes. Read the rest of this entry »

The Cost of a Reject

September 1st, 2016 by Joe Nolan

You’ve heard about the cost of a bug—The longer it takes for a bug to be discovered, the more expensive it is to fix. But have you ever considered the cost of a reject? Do you have visibility into how much rejected stories and bugs are affecting your Scrum team’s capacity in terms of time?

Let’s take a look at just how impactful rejects can be, and how to mitigate them when they occur.

The Cost

Start with an average story or bug ticket in a hypothetical Scrum team:

  • A story or bug ticket is created and assigned to the sprint.
    • The bug has steps to reproduce—if the writer was efficient.
    • For the story, QA and the Scrum team determine the test strategy.
  • The developer works on the ticket and pushes it to QA.
  • QA verifies or rejects the ticket.

Different tickets take different levels of effort. For the sake of this article, let’s assume it takes an average of 30 mins for QA to verify the ticket.

At least, that’s what happens when all goes well. Read the rest of this entry »

How to Avoid Thread.Sleep in Test Automation

August 30th, 2016 by Sahas Subramanian

Several years back, I wanted to understand if it’s really necessary to use thread.sleep in test automation code. Here is what stuck in my mind: Thread.Sleep(n) means blocking the current thread for the number of time slices that occur within “n” milliseconds.

There are a few things to consider before using thread.sleep in your test automation code. They include:

  1. The thread time slice varies from OS to OS and processor to processor. This means that once your thread goes to sleep, it’s going to wait for its turn to come back into the scheduler. So it’s highly likely that the thread.sleep time means more than what you really intended. For example, with thread.sleep(1000), you intended 1,000 milliseconds, but it could potentially sleep for more than 1,000 milliseconds too as it waits for its turn in the scheduler.
  2. Each thread has its own use of CPU and virtual memory. That means our thread acquired its resources (say 1MB virtual memory, some CPU cycles and some context switching cycles), and is simply not using them.
  3. If it’s a foreground thread, it’s preventing the application from exiting as well.
  4. What about Thread.Sleep(0)? This tells the processor that I’m ready to lose the remainder of my time slice, and you may pick up the next thread from the queue. On one hand, it feels good and efficient. But on the other hand, you are also telling the processor that you are better at scheduling than the processor. (I don’t know the side effects this can cause, but I’m not smarter than the processor.)


We are talking about milliseconds when dealing with thread.sleep, but it’s important to understand that the more we use thread.sleep, the more we defer solving the real problem, which is how to handle async in modern web apps. Read the rest of this entry »

The Sauce Journey – Shu Ha Ri

August 25th, 2016 by Joe Alfaro

If you’re attempting to implement an Agile/Scrum development process where none has existed before, you will surely an encounter a moment of frustration on the part of your developers. “Why do we have to do these standups?” “I don’t understand why we need to assign story points, can’t we just get to the projects?” “Where is my technical specification?” Like Ralph Macchio in The Karate Kid, your developers may wonder why you have them doing the engineering equivalent of “wax on, wax off,” when what they really want to do is get into the fight. What Ralph Macchio eventually understands is that the performance of rote, rigid external exercises is a first step on the road to internal mastery, a process well known in the world of martial arts as Shu Ha Ri.

In its broader definitions, Shu Ha Ri describes a process of learning: in the Shu stage, the learner follows directions literally and adheres rigidly to whatever rules the teacher has set. In the Ha stage, the learner begins to see how the rules and directions can be adapted for specific situations, and exercises some judgement in how they should be applied. In the Ri stage, the learner has developed her own techniques, and now innovates freely as the situation demands. Read the rest of this entry »

Building a Test Automation Strategy

August 23rd, 2016 by Chris Riley

Ashley Hunsberger, Greg Sypolt and Chris Riley contributed to this post.

Bringing test automation into your organization is not as easy as writing and running a Selenium script. It involves first getting buy-in, building a team, establishing a strategy, and picking the right tools. During the Q&A portion of a recent webinar hosted by Chris Riley, Ashley Hunsberger, and Greg Sypolt, the presenters realized that these aspects of introducing test automation are well known, but not well understood. In our first post of the series we discussed getting buy-in. Below, in the second post, we discuss how to build a test automation strategy.

Getting started with test automation is easy. If you have a technically minded QA team, you can usually create your test script, sign up for a test cloud, and run the script in just a few hours. But keeping a test automation environment going for the long term is not as easy as any of us would like to believe. QA teams are generally better at building strategy than any other. And when it comes time to build a test automation environment, strategy is a key first element to both getting started and keeping it going. Read the rest of this entry »

Options for Validating HTTP Codes in Selenium

August 18th, 2016 by Alex Entrekin

There is no way to directly get HTTP status codes in the WebDriver API (see the lengthy discussion in issue #141). But that doesn’t mean you have to leave Selenium or go without any status codes in your test scripts.

In fact, some of the supported methods – proxies and tightly coupled headless browsers – should make you feel at home if you’ve transitioned from Selenium RC, or are comfortable with traffic sniffing proxies.

Headless Scriptable WebKits that Play Nicely with Selenium

If you don’t want to invest the time configuring a proxy, consider setting up WebDriver to handle headless testers like HTMLUnitDriver, Jasmine or PhantomJS.HTMLUnitDriver is well-supported in Selenium, but its Javascript engine (Rhino) is not used by most browsers. However, if you know you won’t be testing any pages with Javascript, then it’s no concern.

But if you are, use GhostDriver to enable PhantomJS as WebDriver’s backend. Read the rest of this entry »

Getting Buy-In For Test Automation

August 16th, 2016 by Chris Riley

Ashley Hunsberger, Greg Sypolt and Chris Riley contributed to this post.

Bringing test automation into your organization is not as easy as writing and running a Selenium script. It involves getting buy-in, building a team, establishing a strategy and picking the right tools. During the Q&A portion of a recent webinar hosted by Chris Riley, Ashley Hunsberger, and Greg Sypolt, the presentation team realized that while each aspect is well known, it is not well understood. And that is why in this four-part blog post series, we are going to address each one. This post, the first of the series, discusses getting buy-in.

The need and high-level direction for test automation is usually driven by those who do. In an ideal scenario, both the doers and the decision-makers come to the conclusion together that faster, more repeatable testing is more than just a cost-saver. It supports the initiatives of moving modern development practices forward.

But this is not the reality. Those who know what is possible need to sell the idea from the bottom up. For QA, there are many challenges to doing this: (1) QA teams often do not have their own budget; (2) you are not just convincing development leadership, you are convincing your peers in development and IT as well; (3) all aspects are not in your control (such as continuous integration and test infrastructure). So let’s answer questions asked during the webinar to give some insight on how to approach buy-in. Read the rest of this entry »

A Brief History of the Selenium Testing Framework

August 11th, 2016 by Chris Tozzi

Ever wonder where Selenium (the testing framework, not the mineral you get from eating clams) came from? Here’s a short history of the technology, from its origins more than a decade ago as a proprietary tool through the present era of Webdriver.

ThoughtWorks and The Origins of Selenium

Selenium originated in elder days – by which I mean 2004 – as a tool for testing web applications. It was developed by Jason Huggins, a programmer at ThoughtWorks.

That Selenium originated at ThoughtWorks is interesting. While no one in 2004 was talking about Agile infrastructure, ThoughtWorks was the place where Martin Fowler made his career. Fowler went on to become one of the major thought leaders behind the migration to microservices. While Fowler can’t take credit for Selenium, it seems fitting that the tool, which is an important part of automated testing for DevOps-inspired workflows today, originated in the same place from which the Agile infrastructure revolution later emerged.

Open Source Selenium

At first, Selenium was used only internally by ThoughtWorks employees. But that changed by the end of 2004, when the tool was open-sourced.

Read the rest of this entry »

Sauce Updates OnDemand Plugin for Jenkins Pipeline

August 10th, 2016 by Ken Drachnik

We have updated our OnDemand plugin for Jenkins with explicit support for Jenkins Pipeline. This update enables development and testing teams to easily create, manage, and run automated tests using scripts on Jenkins, resulting in faster testing.

The newly enhanced OnDemand plugin enables Sauce Labs users to skip the Jenkins UI and run tests directly via scripts. Previously, users had to work within the Jenkins UI to create testing tasks. While this is initially convenient, as automation grows and becomes more complex, the process of creating testing tasks becomes more complex, too. Now OnDemand enables testers to automate releases for complex and non-sequential tasks that previously required manual intervention.

Key benefits of the update include:

Create scripts to automate Jenkins operations
Save time by scripting test sequences that can automate testing, retry tests automatically and run processes in parallel. Developers and QA can more easily customize their Jenkins workflows and reporting results.

Automatically create test scripts for Jenkins
Use the Jenkins snippet generator to easily create Pipeline scripts to execute Sauce Commands via Jenkins. Users don’t need to know how to script in Pipeline – simply copy the code snippets to program workflows

The Sauce OnDemand Jenkins Plugin is available as a free integration. Download the plugin from Jenkins and read how to configure it on our docs page.

Happy Testing!

Recap: Selenium and Appium Training Courses from Sauce Labs (Webinar)

August 5th, 2016 by Bill McGee

Thanks to everyone who joined us for our recent webinar, “Selenium and Appium Training Courses from Sauce Labs“, featuring Sauce Labs’ Director of Product Marketing, Ken Drachnik, and Automation Specialist Kevin Berg.

In this webinar, Ken and Kevin announced the availability of our three new technical training programs – Selenium 101, Appium 101, and Sauce Labs Onboarding. They reviewed the course curriculum and class format, and gave a brief demo of the course environment. They also covered classes that are currently in development – Selenium 201, Appium 201 and Sauce Labs for Enterprise.

Interested in learning more about our new training courses? You can find more information on our training website.

Access the recording HERE and view the slides below: