Posts Tagged ‘Cloud’

Automated Testing News: Link Round-Up

January 23rd, 2015 by Bill McGee
Happy Friday! Here’s a quick round-up of some pieces on automated testing, why functional and performance testing should be done simultaneously, the case for Continuous Testing in the Cloud,  how to take your career to the next level, QTP/UFT VS Selenium, and what the top 10 tools were this week on Stackshare.  See below for snippets and links to the full articles.

(more…)

Testing Mobile Apps? Win an iPad!

March 3rd, 2014 by Bill McGee

phoneSauce Labs is all about keeping it real, so naturally we’re adding real devices to our mobile cloud!

So we don’t accidentally participate in an episode of When Keeping It Real Goes Wrong, we’d like to learn more about how you’re doing mobile testing. Help us by filling out this short survey. Uh, did we mention our sweet deal yet? 1 out of every 100 participants will win an iPad Mini. Tell your friends!

Be sure to check back here or follow us on Twitter, Google+, LinkedIn, or Facebook for news about our imminent real devices launch.

Ohai, 100 Million Tests on Sauce Labs!

February 27th, 2014 by Bill McGee

Today we’re ecstatic to announce we surpassed 100 million test runs on Sauce Labs. Yep, that’s right. 100 MILLION tests have been executed on our cloud of real browsers and platforms. Now that’s a whole lotta testing, don’t ya think?!

It’s been an awesome couple years for Sauce, Selenium and automated testing in general. So we thought we’d use this milestone to look back at the growth both internally and within the testing community at large with a fun infographic below. Some of our favorite highlights include:

  • Support for browsers / OS platforms grew from 41 in 2010 to 270+ in 2014
  • Selenium IDE has been downloaded more than 10 million times
  • 10,000+ ounces of Saucy hot sauce distributed all over the world

It’s been a heck of a ride over the last five years, and we couldn’t be happier to be part of the movement. Onward to 200 Million!

Click to enlarge

Click to enlarge

Run Your Selenium Tests Completely in the Cloud Using Travis CI and Sauce Labs

March 18th, 2013 by Santiago Suarez Ordoñez

Passing builds!This is a quick tutorial to get you started running automated tests completely in the cloud as part of your development process using Travis CI and real browsers in Sauce Labs.

As a guinea pig during this blog post, I used a small, open source Python project of my own called cssify. You can find the cssify build results on Travis and even watch running jobs or videos of the browsers in action in the project’s Open Sauce page.

Steps involved:

  • Set up your tests to run on Travis CI
  • Configure Travis CI to start Sauce Connect
  • Configure your Selenium tests to run on Sauce Labs
  • Provide additional details to Sauce

What is Travis CI?

Travis CI logo
Travis CI is a hosted continuous integration service for the open source community. It is integrated with GitHub and offers first class support for:
C, C++, Clojure, Erlang, Go, Groovy, Haskell, Java, JavaScript (with Node.js), Perl, PHP, Python, Ruby and Scala.
Their CI environment provides multiple runtimes (e.g. Node.js or PHP versions), data stores and so on. Because of this, hosting your project on Travis means you can effortlessly test web applications against multiple runtimes and data stores without even having all of them installed locally.

Setup Your Tests to Run on Travis CI

There’s multiple tutorials out there to get started using Travis CI. One of my favorite things about Travis is how easy it is to configure and get started.
Check out the official Travis CI docs to get started in your own programming language.

For running your Selenium tests, remember to install the right dependencies, including the Selenium client bindings, like I did in my project. Also, make sure you make Travis start your webapp before the tests start. Your application needs to be serving pages to be able to test it using our browsers :)

Configure Travis CI to Start Sauce Connect

Sauce Connect creates a magical rainbow tunnel that will allow the Sauce browsers driven by your Selenium tests to access the Web Application that your Travis build starts. Once a Sauce Connect tunnel is running, browsers in our cloud can use it to hit localhost, internal IPs or even to just proxy the public internet through the right location.

I’ve created a small Bash script that takes care of downloading, starting and waiting for Connect to be ready. To use Sauce Connect in your build, all you need to do is curl and run this script in your .travis.yml before_script section as I did in my project:

before_script:
  - curl https://gist.github.com/santiycr/5139565/raw/sauce_connect_setup.sh | bash

Configure Your Selenium Tests to Run on Sauce Labs

Selenium LogoOnce your builds are happily running and Sauce Connect is started automatically by Travis, the next step is having your Selenium scripts run in the Sauce Labs cloud and hit the Application Under Test (AUT) through the tunnel.

To start running real browsers in the Sauce Cloud, make your scripts use Remote WebDriver and point the command executor to localhost:4445 (Sauce Connect will listen in that port and securely and efficiently relay to Sauce). You can find an example of this on the cssify codebase.

self.username = os.environ['SAUCE_USERNAME']
self.key = os.environ['SAUCE_ACCESS_KEY']
hub_url = "%s:%s@localhost:4445" % (self.username, self.key)
self.driver = webdriver.Remote(desired_capabilities=self.caps,
                               command_executor="http://%s/wd/hub" % hub_url)

Now for these browsers to be able to hit the AUT when opening localhost, you need to specify an additional Desired Capability, tunnel-identifier.
The value this capability should be set to is stored in the environment variable TRAVIS_JOB_NUMBER. You can find an example in the cssify codebase for this as well.

self.caps['tunnel-identifier'] = os.environ['TRAVIS_JOB_NUMBER']

Provide Additional Details to Sauce

In addition to running your tests in real browsers using the cloud, there’s additional details you can provide to speed up debugging and navigating test results.
In the cssify tests you can find some examples Desired Capabilities that Sauce uses and that you can pull from Travis CI’s environment variables.

self.caps['build'] = os.environ['TRAVIS_BUILD_NUMBER']
self.caps['tags'] = [os.environ['TRAVIS_PYTHON_VERSION'], 'CI']

That’s it!

You’re set! You are now running your tests using Travis CI, they start a Sauce tunnel on their own and drive real browsers in the cloud. Time to start thinking about running these in parallel to speed up your builds now!

Here are examples of what your Travis CI build should look and what you’ll see from the Sauce side.

The Sauce Partner Program Has Officially Landed

November 21st, 2011 by Ashley Wilson

Sauce’s customer community know us well for our role and enthusiasm in expanding the use of Selenium amongst agile teams big and small. But we’re a young company ourselves and not everyone is aware of us yet. And even among those who do—and might want to use our services—there’s often a need for expertise, capabilities and services that Sauce Labs doesn’t now (or have plans to) offer. So we’re turning to our neighbor communities of facilitators and implementers to:

  • Make it easier to connect customers to service providers and vice versa
  • Expand the use cases of Sauce’s browser cloud service and
  • Show our appreciation for community cooperation in a concrete way

We’re excited to announce the formation of the Sauce Partner program. The program aims to support testing consultants, QA companies, and OEMs by providing infrastructure integrations, demo accounts, commissions, and more, as they enable software development teams to ship quality code faster using Selenium and the Sauce Selenium infrastructure.

The Sauce Partners program contains three different initiatives to cover the full breadth of needs in our expanding community. Read on to find out which one is right for you.

Authorized Partners

Authorized Partners are expertise centers for Selenium and/or Sauce who handle most, if not all, of the end-to-end QA process. This could include test script creation, test maintenance, Selenium implementation, QTP to Selenium training, etc. At Sauce, we get a number of people who write in saying they want to use our service, but only have manual test cases and need a company to write scripts for them. Or they need individualized help implementing Selenium and Sauce Labs. Since we don’t currently support this ourselves, we’re looking for 1-2 companies highly skilled in Selenium and Sauce who will benefit from some brand-name customer referrals. Requirements are that you pass an annual qualification demo and also pay a fee. In addition to the customer referrals, we’ll list you on our website as an authorized partner and also kick back commissions based on sales volume brought in.

Sauce Maître’ D

We designed the Sauce Maître’ D program for individual agile coaches or testing consultancies that help companies implement automation infrastructure, transition manual test practices to automated ones, adopt Selenium, implement Sauce, and more. The Sauce Maître’ D program is free to participant and includes an unlimited demo account for your consulting demonstration purposes. We’ll issue you a promo code to share with your clients that want to use Sauce. Your clients benefit in that your promo code entitles them to a 5% discount off Sauce’s published price schedule.  On a quarterly basis we’ll send you a 10% commission based on the aggregate spending of customers entering your promo code. Simply enroll here, and we’ll set you up.

Sauce OEM

Sauce Labs built the world’s first public browser cloud to meet the needs of agile shops using Selenium to go fast safely.  But now we’re being increasingly approached by OEMs looking to use our browser cloud for applications beyond Selenium and beyond functional testing (such as Security threat injection, deep web data mining, etc.).  So we created the Sauce OEM program to serve the needs of these emerging use cases in addition to those organizations wishing to private-label Sauce’s Selenium service for use with their own customers.

We want to hear from you!

To learn more about and to join any one of these programs, please visit our partner page and register. Or, email us directly at partners@saucelabs.com. We look forward to partnering with you!

Security Through Purity — Bypassing the Red Queen

September 27th, 2011 by The Sauce Labs Team

As a cloud service, Sauce Labs uses quite a few virtual machines to run our browsers. With Windows images taking about 15 minutes to boot in EC2 and other cloud providers, the most efficient way to use these resources is to cleanup between customers and reuse. Unfortunately, reusing VMs paves a path for malicious users to install software that will continue to run while other customers use them. This means sensitive data could be captured and sent to a 3rd party or, even worse, a 3rd party could gain access to a customer’s internal network if they are testing with a tunneling service.[1. Did you know you can use Sauce Labs to test your internal network? Check out Sauce Connect for secure way to use our cloud service to test your internal infrastructure. It’s been vetted by a well-known financial institution, a well-known domestic airline, and a variety of other high-profile customers.] How can a VM be protected from malicious software installation?

There are a variety of tools one can use to lock down Windows. The tools provided by Windows itself are the Software Restriction Policy and AppLocker technologies. And a cursory Google search will reveal current dll load hijacking methods for getting around them. The reality is it’s a Red Queen race between securing a system and finding a new exploit — “It takes all the running you can do, to keep in the same place.” For all intents and purposes, it’s like submitting your laptop to a hacking contest. Is there anything you could do to feel comfortable using it again?

the only winning move is not to play

Like Joshua learned, “the only winning move is not to play,” and that’s how we roll at Sauce Labs. Every single session, whether you’re running manual or OnDemand, gets a fresh VM that’s never been used by anyone else.[3. In this way, Sauce Labs has lunched nearly 1.3 million VMs in the last 30 days.] At the end of your session — whether it’s 10 minutes or 10 seconds — the VM (with all its data) is completely destroyed. By avoiding the unsolvable problem of guaranteeing a system secure across anonymous use, we instead guarantee a system that has never and will never be used by any customer besides you.[2. Since we give you a fresh VM in under 20 seconds, a 15 minute boot time would require us to keep a lot of expensive, idle resources around to handle demand spikes. That’s why we developed a cloud technology which allows us to cold boot VMs in under 40 seconds. Aside from meaning demand spikes are handled with minimal delay, this also means there is no additional expense for providing the best security.]

A desirable side-effect of the Security Through Purity model is increased reliability. By always using pristine VMs for every session, every session is run with the same settings. This avoids intermittent failures that arise from the OS being in a state dependent on what the previous users did.

At Sauce Labs core is the ability to offer secure, remote browsers for interactive, mobile, JS, and Selenium testing to our customers. Our attention to security provides confidence you can test any part of your external or internal services without having to build and manage your own browser testing infrastructure.

Sauce Mac OS X Private Beta Signup!

August 23rd, 2011 by Jason Huggins

From the creation of Sauce OnDemand to the recent release of Sauce Scout, the most popular feature request from users has been OS X support. Well, we’re stoked to say that the wait is almost over. We are readying a very early version of Mac OS X support in our browser testing cloud.

(In case you’re wondering, we will resist the incredible temptation to call this new offering “apple sauce”).

If you’re willing to give us tons of feedback, and will be patient with growing pains, please sign up!

Like the rest of our cloud testing services, each Mac OS X session will include the browser testing features you’re familiar with. You’ll get a video of each session, and can take screenshots at any time. With every new test session, you get a clean-slate, freshly booted machine to control. And when the session is over, we delete the virtual machine, so there’s no lingering data left behind on the machine to worry about.

The Mac OS X private beta will initially be limited due to available systems and we’ll contact you as resourcing permits. First priority will be given to corporate users.

In you’re interested in joining the beta, complete the sign-up form, and we’ll email you soon with details.

JavaScript Unit Testing with Jellyfish and OnDemand

June 20th, 2011 by Adam Christian

Jellyfish logo

Web application testing turns out to be very similar to an onion. You keep peeling back layers and finding more testing that can, and probably should, be done. But you shouldn’t only be concerned with testing the smooth outer layer that your users will see.  There’s still plenty of automated testing to be done on those inner layers, and it doesn’t require much extra work.

The testing I’m referring to is the JavaScript unit tests. For most development teams, this is an incredibly important step and should be done well before using Selenium for testing web UI and user flows.

Most people think about Selenium as the “click” here, “type” there project. And for many intents and purposes, that’s correct. However, Selenium 2 is offering some new functionality that lends itself to a slightly bigger set of use cases. What I’m talking about is the ‘execute’ method. This allows you to execute arbitrary JavaScript in the AUT (application under test) scope, without having to deal with the frustrating complexities of ‘getEval’ in Selenium 1.

Specifically because of this functionality, I wrote a driver for Jellyfish (jelly.io), a framework that allows you to write node .js scripts that run JavaScript in the browser. In order to do this, I implemented a tiny subset of the WebDriver JSON wire protocol in node.js and called it ‘wd‘. These projects are interesting all in their own right, but to get straight to the point: If you have tests in FooUnit, QUnit or any other JavaScript testing framework that run in the browser, we now have a way for you to do all of that in the Sauce OnDemand cloud.

jelly_integration

For this example, lets use QUnit. In the hopes of covering all the bases, lets assume your QUnit tests are hosted on an internal web site to your company. Lets also assume that the way you currently run them is by opening a browser window and going to ‘file open’, or navigating to a local URL and sitting there watching the tests run. When they are done, you’ll manually look through the tests and results to decide if you broke things or not.

With Jellyfish and OnDemand, the way this now works is that you have a node.js script that runs after every check-in (perhaps by Jenkins) using different browser and platform combinations. When the tests are complete, an email will be sent to all your developers each time you have a failure.

architecture

So how do we do this?

Start by installing Jellyfish, the directions for which are front and center at jelly.io.

Next we will build a straightforward Jellyfish script that will start a Sauce Labs session, navigate to your QUnit tests URL, and assert the results when they finish.

For my examples, I am going to be using the full ‘JQuery Test Suite’ that I got from ‘jquery.com’.

Lets talk about the contents of the script one piece at a time:

var jellyfish = require('jellyfish')  
  , assert = require('assert');

 

This imports the jellyfish package and the assert module, which is a nice lightweight way to make assertions in node.js.

var sauce = jellyfish.createSauce();
sauce.opts.platform = "VISTA";
sauce.opts.browserName = "iexplorer";
sauce.opts.version = "9";

 

This instantiates a new Jellyfish object and lets it know that this one will be talking to Sauce Labs. The username and password for your Sauce Labs account can be passed through here directly, but in this case, I’m storing it in my ~/.jfrc file –, which is nice for simplicity. Next we’ll specify the platform, browser, and version we want to use to run our test (more information about .jfrc can be found here).

sauce.on('result', function(res) {  
  console.log(sauce.name + ' : '+sauce.tid + ' - \x1b[33m%s\x1b[0m', JSON.stringify(res));
});

 

This one is optional, but isn’t it nice to get well-formatted output in your logs?

sauce.on('complete', function(res) {  
  console.log(sauce.name + ' : '+sauce.tid + ' - \x1b[33m%s\x1b[0m', JSON.stringify(res));
  sauce.js("window.testOutput", function(o) {    
    try {      
      assert.equal(o.result.failed, 0);      
      sauce.stop(function() {      
        process.exit(0);    
      })  
    }    
    catch(e) {      
      console.log(e);      
      sauce.stop(function() {      
        process.exit(1);      
      })    
    }  
  })
});

 

This listener is very important since generally our communication with Selenium on Sauce OnDemand is one directional. However, that has been abstracted away. During the script, Jellyfish will poll Selenium every few seconds and ask ‘has window.jfComplete been set to true’? If so, an event will be triggered called ‘complete’. In the next section, you will see how the results are set, but at this point, I just have Jellyfish ask the page for the contents of ‘window.testOutput’. Next we assert the value of the number of failures. In this case, I am catching that exception so that I can print it to the terminal and exit the script with the correct return code for CI.

sauce.go("http://myurl.com/jquery/test/index.html")  
  .js("QUnit.done = function(res) {
    window.testOutput=res;
    window.jfComplete=true;
   }");

 

This is where the magic happens. This code navigates to the URL of the QUnit tests that start automatically, and as soon as we have navigated there, I redefine the QUnit done event to set the window.jfComplete flag. Then I set the results output of the test run in a place I can access it in the above code.

Usually the first question asked here is how to access QUnit tests that are on an internal network. Fortunately, we have a product at Sauce Labs called Sauce Connect that allows our cloud’s VMs to securely access whatever your test machines access on the internal network. It is incredibly simple to setup and you can do so by following the comprehensive documentation.

After that tunnel is running (you can just leave it running), you can now update the start URL in ‘sauce.go’ to reflect the URL you use internally to access your tests in the browser.

Jenkins has hooks for essentially any deployment platform or source control repository, and it also has post-deploy hooks that can be configured to send you an email. You can also use either a ‘matrix project’ or environment variables to make one job run your test on all of our available platform and browser combinations.

The example script can be downloaded here from github and I also made one of my Jellyfish QUnit test runs public for your viewing pleasure. Happy testing!

Sauce Builder Webinar: From zero to creating, storing and running automated tests in under 30 mins

May 3rd, 2011 by The Sauce Labs Team


As you may have heard, we recently released a new, free tool that makes building and running Selenium tests easier than before. Sauce Builder is now available for download, and with it comes the ability to create Selenium tests in your Firefox browser by simply clicking around your application. You can then run those tests in Sauce OnDemand with just a few clicks, no Selenium expertise necessary.

To explain more about Sauce Builder and how automated testing can accelerate your development velocity, we held a webinar with Adam Christian, a Sauce Labs developer and project lead for Sauce Builder.

With Sauce Builder, we’re hoping to simplify and improve the cross-browser testing process for development and QA teams here is an updated video that will demonstrate how to use

  • Build Selenium tests with zero programming – Simply click through your application and Sauce Builder writes a Selenium scripts that reflect your actions.
  • Export results in the language of your choice – HTML, JAVA, Groovy, C#, Perl, PHP, Python and Ruby so your tests speak the same language as your application and dev team.
  • Eliminate bugs faster – Use immediate video playback of your tests in action, and share them with your teammates.
  • Remove test infrastructure headaches – Sauce Builder makes it a snap to either run tests locally in Firefox or in the cloud with access to all the browser / OS combinations supported in the super scalable Sauce OnDemand service.

Watch an updated video presentation that will show you how to build a CI system using Selenium Builder, GitHub, Travis and Sauce Labs.

 

Testing Your Mobile Apps with Selenium 2 and Frank

March 30th, 2011 by Ashley Wilson

In case you missed our last SF Selenium Meetup, check out video below of the two great presentations on mobile testing. Pete Hodgsen, a software consultant with Thoughtworks, discussed how to use Frank, a new testing framework he’s involved in developing, for automated acceptance testing of native iOS apps. Jason Huggins, creator of Selenium and co-founder of Sauce Labs, followed this by demoing how to test Android and iPhone applications using Selenium 2.

Many thanks to our friends at CoTweet for co-hosting and making this last meetup such a success! If you think your company might be interested in hosting a future meetup, please get in touch. Our next meetup will be in early May at Mozilla (we’re skipping April because of the Selenium Conference happening next week). Stay tuned for more details!