We recently added a new feature to our manual browser testing web client that we’re excited about. The “Live Share” feature in Manual Sauce, our interactive testing tool, allows you to share your live browser sessions with coworkers simply by sending a link. You can collaborate using your favorite IM client to demonstrate the exact steps to reproduce a bug you’ve run into and interact with the browser running on Sauce at the same time!
To try this out, first start a manual testing session. Once your browser has loaded, look to the header bar for the live share button next to the URL bar. Hover over the button to pop open a dialogue box that lets you specify the email address of the person you want to receive a private, unique link for sharing the manual test session. Then click the share button.
If you leave the email boxes empty and click the share button, no email will be sent but your test’s visibility will change to ‘share’. This means anyone with the link can access the test session and view it live. If it’s easier to share the session directly via Gmail or whatever email client you prefer, just copy the link from the live sharing dialog and send it to your coworker(s).
To indicate that the tests can by accessed by anyone with proper link, the sharing button will be turned on next to the address bar. You can click this button to turn off live sharing mode. Each time you click that button, it makes a call to our REST API and the test visibility mode is updated (http://saucelabs.com/docs/rest).
Please note that by default, the job visibility mode is set to ‘private’. This way, only the owner of the test can access it. When you click the button, the mode is set to ‘share’, allowing everyone with valid secret link to access the test page.
We hope you find this new feature useful. And as usual, let us know if you have any questions. Happy testing!
This is a quick tutorial to get you started running automated tests completely in the cloud as part of your development process using Travis CI and real browsers in Sauce Labs.
As a guinea pig during this blog post, I used a small, open source Python project of my own called cssify. You can find the cssify build results on Travis and even watch running jobs or videos of the browsers in action in the project’s Open Sauce page.
Set up your tests to run on Travis CI
Configure Travis CI to start Sauce Connect
Configure your Selenium tests to run on Sauce Labs
Provide additional details to Sauce
What is Travis CI?
Travis CI is a hosted continuous integration service for the open source community. It is integrated with GitHub and offers first class support for:
Their CI environment provides multiple runtimes (e.g. Node.js or PHP versions), data stores and so on. Because of this, hosting your project on Travis means you can effortlessly test web applications against multiple runtimes and data stores without even having all of them installed locally.
Setup Your Tests to Run on Travis CI
There’s multiple tutorials out there to get started using Travis CI. One of my favorite things about Travis is how easy it is to configure and get started.
Check out the official Travis CI docs to get started in your own programming language.
For running your Selenium tests, remember to install the right dependencies, including the Selenium client bindings, like I did in my project. Also, make sure you make Travis start your webapp before the tests start. Your application needs to be serving pages to be able to test it using our browsers :)
Configure Travis CI to Start Sauce Connect
Sauce Connect creates a magical rainbow tunnel that will allow the Sauce browsers driven by your Selenium tests to access the Web Application that your Travis build starts. Once a Sauce Connect tunnel is running, browsers in our cloud can use it to hit localhost, internal IPs or even to just proxy the public internet through the right location.
I’ve created a small Bash script that takes care of downloading, starting and waiting for Connect to be ready. To use Sauce Connect in your build, all you need to do is curl and run this script in your .travis.yml before_script section as I did in my project:
Configure Your Selenium Tests to Run on Sauce Labs
Once your builds are happily running and Sauce Connect is started automatically by Travis, the next step is having your Selenium scripts run in the Sauce Labs cloud and hit the Application Under Test (AUT) through the tunnel.
To start running real browsers in the Sauce Cloud, make your scripts use Remote WebDriver and point the command executor to localhost:4445 (Sauce Connect will listen in that port and securely and efficiently relay to Sauce). You can find an example of this on the cssify codebase.
Now for these browsers to be able to hit the AUT when opening localhost, you need to specify an additional Desired Capability, tunnel-identifier.
The value this capability should be set to is stored in the environment variable TRAVIS_JOB_NUMBER. You can find an example in the cssify codebase for this as well.
In addition to running your tests in real browsers using the cloud, there’s additional details you can provide to speed up debugging and navigating test results.
In the cssify tests you can find some examples Desired Capabilities that Sauce uses and that you can pull from Travis CI’s environment variables.
You’re set! You are now running your tests using Travis CI, they start a Sauce tunnel on their own and drive real browsers in the cloud. Time to start thinking about running these in parallel to speed up your builds now!
A group of Saucers are heading to Santa Clara today to attend PyCon, the largest annual gathering of Python users. Given that we’re a Python shop and it’s our first time sponsoring the event, we’re especially excited to hang with folks and chat about Sauce & Selenium.
Stop by booth #178 tonight from 6-8pm as well as between 10am-5pm Friday & Saturday to talk about automated testing. OR come by to get a look at our favorite robot being both functional and fun as it tests real mobile devices and plays Angry Birds. You’ll also be able to enter in the raffle to win said robot and try various Saucy dips at our salsa tasting station.
Integrating Selenium and Sauce Labs into your Continuous Integration Build
CI can be hard. Making Selenium a part of your CI infrastructure is even harder. This tutorial will cover the end-to-end process of adding Selenium infrastructure to your CI build using Sauce Labs, and integrations with Jenkins and Travis CI. Specific attention will be paid to common pain points, hardening your tests against common failures, and features that are provided by Sauce Labs’ integration with Travis CI.
And Jasonwill also give a main conference talk on Saturday at 12:10pm:
Mobile Application Testing with Python and Selenium
Selenium has grown to be a mature platform on the desktop, but with ‘mobile now’ being the mantra for so many companies, can we use Selenium to effectively test mobile apps? What about Native apps? This talk will cover using Python to test mobile web applications with Selenium, as well as an in depth overview of the future of Selenium to test Native iOS and Android applications.
We know change is hard, particularly when you’re used to working with a certain tool. That’s why we want to make the switch from BrowserLab to Sauce Labs as seamless as possible. For the next 30 days, you can receive 10 hours of free testing on our service. To redeem this offer, sign up for a free account at http://saucelabs.com/signup, pick the free plan option, and enter in the promo code adobe.
To learn more about Sauce and our offering, read on as we first explore the similarities between the two services. Next we’ll dive into the enhanced features that set us apart from other interactive testing tools out there. Finally, we’ll go through a quick tutorial to get you testing on Sauce and start your free testing time!
So, what’s similar?
Like BrowserLab, our service exists in the cloud so you never have to worry about setting up infrastructure. From Internet Explorer to Firefox to Chrome, we have all the major browsers covered for for Windows, Linux, Mac OS X. We even have mobile browsers on iOS & Android covered.
With Sauce, you can take as many screenshots as you like while exploring how your site looks across various browsers. Every screenshot gets stored on your personal account page so you can access and share them later with colleagues and friends.
With Sauce Connect, our secure tunneling solution, you can easily test local sites or ones behind a firewall. You can read more about Sauce Connect here.
And what’s different?
Each time you open a site on Sauce, we give you control of a real, live browser in our cloud to navigate your site. This not only provides visual confirmation that your website or web app looks how you were expecting, but that it’s behaving correctly too. Your session will be free of left over cookies or junky data to confuse the testing process – just a nice clean machine and browser for you to test on.
Extensive browser support
Testing on Sauce means instant access to more 100+ OS and Browser combinations – including desktop and mobile browsers. When a new browser is released, it’s available in our service immediately. We also have a wide array of legacy browsers, including antiques like Firefox 3.0 and IE 6.
Every browser in our cloud comes equipped with the standard debugging tools, like Firebug and the IE developer toolkit. Additionally, we provide the ability to share your live test session with a co-worker for realtime, collaborative debugging.
As you test your site on Sauce, a video is recorded in the background every time. We find this comes in handy when the time comes to share bugs with the rest of your team – they can really see how the site responds and looks for an end user. To access the video, just close your session and visit your account page to see a list of all your videos and screenshots.
Sounds great, right? So how do you get started?
The first step is to sign up for a free account at http://saucelabs.com/signup. While signing up, click on “I have a promo code” and enter the promo code “adobe” (case-sensitive) to get a bonus 10 hours (600 minutes) of free testing. Once your bonus runs out, you can continue on the free plan (30 minutes of free testing a month) or sign up for the $12 a month plan where you will get unlimited manual testing minutes.
In addition, if you’re on a Mac, we suggest trying out Sauce for Mac, our native desktop app for cross-browser testing. Designed with the front-end developer and web designer in mind, you can command-tab across multiple browsers and OSs simultaneously. To download Sauce for Mac and get started, check out this blog post.
If you’re a PC user or prefer to use a browser from your Mac, after you have signed up and are logged into your account, you can launch a new interactive session right from your account page by clicking on “New Interactive Session.” A box will pop up prompting you to enter your URL and select a browser and OS.
Next you’ll see a page letting you know we’re spinning up your browser. Every time you test on Sauce, we provision a new, pristine virtual machine (VM) for you. When you’re done testing, we destroy that VM, ensuring no data or cookies are ever stored or shared between users and sessions. (Read more about our careful approach to security).
You are now in full control of the browser to start interactively testing your website!
You can take a screenshot, log an issue, or even share the live session with another person for interactive debugging:
Today I’m going to show you how you can use Selenium Builder with GitHub, Travis, and Sauce Labs to make a continuous integration solution for your website that’s all on the Internet, nothing stored locally. You can either follow along with the demo video below or read on for the written tutorial.
The great thing about this setup is that if you put your site code and your tests into the same GitHub repository, then whenever you update the site code, and whenever you update the tests, Travis will rerun your Selenium tests on Sauce OnDemand – and then send you an email about whether or not they still work!
So you can get a full Selenium CI solution by putting together these components. And because they’re all components, if you want to use something else in part – for example Jenkins instead of Travis – you can construct your own solution.
For now I’m going to show you this set of four things that work really well together. The great thing about this is that you store everything on GitHub, so all your tests are stored in the cloud, nothing is stored on some local machine somewhere. You can record and edit tests from any computer that runs Firefox. You can access your tests, edit them, & play them back locally. And all of the actual execution of the testing – the CI – is handled by Travis and Sauce. So you actually don’t need to build any of your own testing infrastructure!
OK! So how do you actually do this?
The first thing you need to do is install the plugins. You go into Selenium Builder, you go into Plugins, and you just click install on GitHub integration and Sauce. Restart Builder.
Next up, you want to record some tests. Let’s just say for now we’re testing this website.
Let’s record a verification.
Now we’re going to save this test. In the File menu, save the test to GitHub.
Because this is the first time we’re using it, I’m going to have to enter my details.
Now, as you can see, we can access our repositories. And I’ve set up this little repo here as an example one.
So let’s save this test in here, as JSON, which is the standard format Builder works in. This means we can later open it again and edit it.
Let’s do another test.
Now we save that one as well. Just record as many tests as you want and put them all into this place.
Now we’ve done that, the next thing we need to do is configure the Selenium interpreter, which is actually a little node.js program that takes these JSON scripts and uses Webdriver to play them back.
Let’s clone the repo to disk so we can add the config files we need.
Create a new text file. I’ve just pasted in a bunch of things, which are actually reasonably straightforward.
The config is a JSON file that contains two things: the settings that you want to run the tests (in this case, we want to run them on Firefox) and if you want to run them on Sauce Labs’ infrastructure. You have to tell Sauce your username and access key before this can happen, but it’s not a good idea to commit your Sauce access key to a GitHub repository.
So instead, we’re going to put the username and access key into environment variables. As long as we run the interpreter with the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables set to the right values, it will work fine. And how we set those variables, we’ll get to in a moment, as that will be in the Travis configuration.
Here we have the tests we actually want to run. This config format supports glob syntax, so you don’t have to list each test individually. Just put in the path to the directory with an asterisk at the end, and it will find the tests. And it will ignore anything that’s not a JSON file.
That’s the entirety of the config, so let’s save that in our repo.
And commit and push it.
Now the next thing we need to do is to tell Travis that we exist, and that we would like to use its services.
You can log yourself in…
…using your GitHub account.
And then you go in here…
…and you enable Travis integration for this particular repo.
OK. Now the final step is to configure Travis. We need to tell Travis what running an integration test for this particular project actually means. That’s done by putting a file into the top level of the repository, which tells Travis what to do. It’s called .travis.yml.
So again, I’ve just pasted in something.
Now we tell it to install the Selenium interpreter, and then run it with the configuration we already made. And the one thing we’re lacking here is the environment variable for the Sauce access key. Setting variables in Travis is easy if you don’t mind them being public, which you probably don’t mind for your Sauce username.
Now this is the one tricky bit in this tutorial. You need to encrypt your access key to let Travis know what it is without revealing it to the world. The way to do that is to first install a Ruby gem.
Once that’s installed, make sure you’re in the repository you want to encrypt something for and log in to Travis.
Then you can use this command to encrypt an environment variable.
What comes out is a blob of encrypted data…
…which you can paste into the .travis.yml config file like this. The encrypted data is tied to your git credentials and this particular repo, so it should be safe from snooping or theft. Your access key should also be safe. If it does ever become compromised, you can also generate a new one. That’s what I’ll be doing in a few minutes.
Now you can save this file…
…commit it, and push it. And of course this means Travis now goes “Oh, this has been pushed to! OK, is there a .travis.yml? Yeah, there is.”
Let’s actually go onto Travis. We can see it’s in the queue right now.
Ok, good, it’s running the tests.
You can see the details of each test being run.
But you don’t need to know any of the details, because in the end, we get an email saying, “Hey! It’s succeeded, everything’s fine!”
So I’ll just show you what happens if you break things. You can edit these JSON scripts with a text editor, which is pretty easy.
Let’s go in here and change this verify to something that’s not actually the case.
Save, commit, and push, and Travis runs again.
And now we get told that the build is broken! One of our tests failed!
So let’s fix the test up again.
Save, commit, push.
And shortly after, Travis tells us that we’ve fixed the build. Excellent.
That’s all really – you can use these four components to make a continuous integration system for your site. If you have any questions, drop Adam or me an email.
Since the first release of Selenium Builder two years ago, we’ve been focusing on extending its capabilities and improving the underlying code. For those not familiar, Selenium Builder is an open source tool that helps you easily build and export Selenium tests using record and playback capabilities, and then export those scripts to any programming language.
After many months in the wild and tons of great feedback from the community (plus pull requests and patches on Github), we are incredibly happy to announce the release of Selenium Builder v.2.0 and an updated version of Sauce for Selenium Builder.
Watch the video below to see what’s new and read on for more details.
Selenium 2/WebDriver Support
The most important feature to come along in Selenium Builder is built-in support for Selenium 2/Webdriver. You can record, edit, and play back test scripts based on the new Selenium version, which is particularly handy for folks needing help making the switch from Selenium RC to WebDriver. Plus, since Builder uses the same interface for this as it does for Selenium 1, you can use a mixture of both technologies to build your scripts. Selenium Builder can also translate between the two versions with reasonable accuracy.
Additionally, you can export Selenium 2 scripts into a wide variety of languages: Java, Ruby, Python, PHP, Node.js, and C# (with more language support to follow). Se Builder 2 also defines a clean JSON-based format for storing scripts for later editing. These JSON-based scripts can be played back by a standalone interpreter, so you don’t have to commit to a particular programming language for playback.
There’s another exciting addition to Selenium Builder and that’s support for plugins that extend Builder’s functionality. Plugins can be installed and managed directly from the user interface of Builder. The two plugins listed below are available in this release, with more on the way!
Sauce for Selenium Builder Integration
With this plugin, you can run scripts on Sauce Labs directly from Builder’s interface. Simply export your tests to the programming language of choice and pick from the 100+ browser/OS combos we support to run against. To download Selenium Builder with the Sauce capabilities already baked in, click here.
This plugin lets you store your scripts on GitHub for safe storage and easy collaboration.
Foreign Language Support
And last, but certainly not least, Selenium Builder is now available in German and French (as well as English).
If you’d like to improve Selenium Builder by adding new languages, creating a plugin, or just fixing issues, have a look at the documentation and drop us a line!
You’re probably wasting two things when you’re testing. Your time… because the other thing is your extra CPUs. Any time you’re waiting on a relatively slow resource, your CPU is just sitting there, twiddling its silicon thumbs. If you’re only using one CPU core at a time, the other cores are doing much the same. Unfortunately for web developers who do things involving CRUD operations, slow resources includes databases. Most unfortunately for you, Dear Reader (and us, Dear… Us), they also include the browsers you’re integrating with for Selenium testing.
This is what waiting for browsers and io should be measured with
One of the best ways to get more test for your tokens is to run more tests at once. If you’ve got several tests going, even if some are waiting for one slow resource, the others can use the CPU. The more tests you can run at once, the shorter your test cycles will be, especially if you could, say, spin up more then one copy of the slow resource to test with (Psst: I’m talking about Sauce Labs Parallelization).
I’ve been looking for ways to make parallelization easier when testing with Ruby, and I stumbled across the Parallel Tests gem. It’s actively developed, has some nice documentation and integrates with rspec, rspec-rails and test:unit. Their benchmarks show that using the gem, the Rails ActionPack test suite time was cut in half, from 88 seconds to 44, with just 4 test runners. This, conveniently, is the number of parallel tests you can run with a Mild plan.
Checkmate. Wait. Snap. Game Set Match?
So for many of you, there’s already a possibility of making your tests take half the time. Which means you can run twice as many. Which means a much faster turnaround time for TDD, BDD, and release testing.
After a considerable period of investigation and bug fixing in collaboration with the Selenium project and the Legion of the Bouncy Castle, we are happy to announce we’re moving to Selenium 2.30.0 as the default version in our service. You can find more details about this release in the official changelog.
This transition is scheduled for Friday, March 1st. In the meantime, we advise you to try your tests on this new version by adding the following key-value to your tests’ Desired Capabilities:
If you see any issues after moving over to this new version, we definitely want to hear about it. And remember, once we move everyone to 2.30.0, you can still keep your tests on our previous default (Selenium 2.18.0) version by using the “selenium-version” capability outlined above.
The following is a post by Dylan Lacey. Kinda. He’s chosen to do it in interpretive dance easily digestible image format.
This is San Francisco.
I was recently there…
…because I am the new Ruby Developer Evangelist at Sauce Labs.
My job involves taking a fair few of these;
And sharing a lot of these. It’s a burden.
I’m here to help get Ruby developers up and running, helping them to Drink the Sauce.
It’s SO awesome.
I’m insanely thrilled to be working with such smart people on an amazing product!
My bailiwick is to make it better for Ruby developers to use Sauce Labs’ stuff, including improving the gems, writing better documentation and building the community. Plus, it’s a unique opportunity to use my personality to insult people all over the world! If you want to offend your manager, disrupt your office and get kicked out of your favorite bar for getting shouty about whether RSpec is better than Test::Unit, let me know. And if you want help with the Sauce Gem or tests from Ruby land, hit me up.
If you’re in SF this afternoon, come have a cup of coffee on Sauce Labs and Travis CI!
Josh (@j2h) of Travis CI is in San Francisco till the end of the week, and we’ll be co-hosting a coffee office hours with him today from 2pm today at Sightglass on 270 7th Street.
We’ll be available to chat about anything and everything Sauce and Travis related. We’re always happy to show off our open source work on Appium and to get more open source projects set up to do their JS unit testing and Selenium testing with Open Sauce and Travis CI.