How To Scale Your Tests
If you’ve ever needed to test features in an older browser like Internet Explorer 8 then odds are you ran a virtual machine (VM) on your computer with a “legit” version of Windows XP.
Handy, but what happens when you need to check things on multiple versions of IE? Now you’re looking at multiple VMs. And what about when you need to scale and cover other browser and Operating System (OS) combinations? Now you’re looking at provisioning, running, and maintaining your own farm of machines and standing up something like Selenium Grid to coordinate tests across them.
And all you wanted to do was run your tests on the browsers you cared about…
Rather than take on the overhead of a test infrastructure you can easily outsource things to a third-party cloud provider – like Sauce Labs
A Selenium Remote, Selenium Grid, And Sauce Labs Primer
At the heart of Selenium at scale is the use of Selenium Grid and Selenium Remote. Selenium Grid lets you distribute test execution across several machines and you connect to it with Selenium Remote – specifying the browser type and version through the use of Selenium Remote’s
This is fundamentally how Sauce Labs works. Behind the curtain they are ultimately running Selenium Grid, and they receive and execute your tests through Selenium Remote – knowing which browser and operating system to use because of the
Capabilities its users specify.
Let’s dig in with an example.
Part 1: Initial Setup
NOTE: You’ll need an account to use Sauce Labs. Their free one offers enough to get you started.
With Sauce Labs we need to provide specifics about what we want in our test environment, our credentials, and configure Selenium a little bit differently than we have been. Building on the last write-up, let’s start by creating a new config file for cloud execution (‘config_cloud.rb’).
ENV['base_url'] ||= 'http://the-internet.herokuapp.com'
ENV['host'] = 'saucelabs'
ENV['operating_system'] ||= 'Windows XP'
ENV['browser'] ||= 'internet_explorer'
ENV['browser_version'] ||= '8'
ENV['SAUCE_USERNAME'] ||= 'your-sauce-username'
ENV['SAUCE_ACCESS_KEY'] ||= 'your-sauce-access-key'
Notice the use of a host environment variable. This is what we’ll use in our
spec_helper file to determine whether to run things locally or in the cloud — and we’ll use the other environment variables to populate the
RSpec.configure do |config|
caps = Selenium::WebDriver::Remote::Capabilities.send(ENV['browser'])
caps.version = ENV['browser_version']
caps.platform = ENV['operating_system']
caps[:name] = example.metadata[:full_description]
@driver = Selenium::WebDriver.for(
@driver = Selenium::WebDriver.for :firefox
Notice that we’ve added a conditional to check on the host environment variable. If the host is set to ‘saucelabs’, then we configure the capabilities for Selenium Remote, passing in the requisite information that we will need for our Sauce Labs session. Otherwise, it will run our tests locally using Firefox.
Now if we run our test suite (
rspec -r ./config_cloud.rb) and navigate to our Sauce Labs Account page then we should see each of the tests running in their own job, with proper names, against Internet Explorer 8.
Part 2: Test Status
The only thing missing now is the pass/fail status of the job. In our local terminal window everything should be coming up green. But in the list of our Sauce jobs, the ‘Results’ panel for everything will just say ‘Finished’. This will make our results less useful in the long run, so let’s fix it.
Thanks to Sauce Labs’ sauce whisk gem, it’s really simple to do.
After we install it we will need to require it somewhere, and our ‘config_cloud.rb’ file seems like a logical place, since we will only need it when running our tests in Sauce. So let’s add it to the top of the file.
require 'sauce_whisk' ...
All that’s left is to add an action to our
after(:each) block in our ‘spec_helper’ file.
Before we issue
@driver.quit we will want to grab the job ID from our
@driver object and set the job status based on the test result. Also, we’ll want to make sure that it only executes when running tests against Sauce Labs — so we’ll want to wrap it in a conditional check against the host environment variable.
RSpec.configure do |config|
if ENV['host'] == 'saucelabs'
Now when we run our tests (
rspec -r ./config_cloud.rb) and navigate to our Sauce Labs Account page, we should see our tests running like before – but now when they finish there should be a proper test status (e.g., ‘Pass’ or ‘Fail’).
Speeding Up Your Tests
Now we can easily run our tests in Sauce Labs, but it’s a real bummer that all of our tests are executing in series. As our suite grows, things will quickly start to add up and really hamper our ability to get feedback quickly.
With parallelization we can quickly remedy this — and there are a few ways to go about accomplishing it.
- In code
- Through a test runner
- Through your Continuous Integration (CI) Server
For a deeper dive on how to scale your tests (e.g., running tests in parallel, testing applications behind a firewall, outputting Sauce Labs job information into your test results, etc.) then be sure to check out The Selenium Guidebook.
Up next, in my final write-up, I’ll help you tie everything together through the use of Continuous Integration.