Posts Tagged ‘Cloud’

Manual Testing In The Cloud (Beta Program!)

February 17th, 2011 by The Sauce Labs Team

We’re pleased to announce a new private beta program for manual exploratory testing that allows you to instantly control any browser, in the cloud, using your mouse and keyboard.

With this new service, each time you find a bug,  you’ll have access to the video and screenshots of every step. You can record video of your tests and track test time by browser. Plus, sharing the bug report with the rest of your team is a snap, as we provide a convenient way to securely integrate with your existing bug tracker.

Click here to join the beta!

Why jQuery in Selenium CSS Locators Is The Way To Go

January 31st, 2011 by Santiago Suarez Ordoñez

You may not have heard about this, but a while ago, Jason Huggins moved Selenium 1’s CSS locator engine from CSSQuery to Sizzle, jQuery’s CSS selector library.

While this may not sound like a big deal to most users, it actually is, and in this blog post, I’ll explain why and how to start using all the cool features that come with this change.

Update: For those of you using or willing to use Selenium 2/Webdriver, you may want to re-consider the special Sizzle pseudoclases like :contains or :nth that I’m talking about in this blog post. For technical reasons, injecting Sizzle in browsers driven by Selenium 2 is not as cheap as with Selenium 1, so the Devs have decided to rely on the browsers’ implementation of CSS (standard CSS only) and fallback to Sizzle when needed (in case of old browsers which don’t provide native css selecting for JavaScript). In Selenium 2 land, my advice, sadly, is to stick to the standard and avoid most of these shortcuts :,(

Why is Sizzle awesome?

Well, Sizzle is jQuery‘s selector engine, and that means A LOT. For those of you who don’t know, jQuery is the javascript library used by almost 30 percent of all the websites. (As in, “30 percent of the whole effing Internet!”).

Because of that, Sizzle gets an insane amount of usage, and therefore, testing. Sizzle’s code is used by an average of 1 site for every 3 you visit. Its codebase has over a thousand followers on github and more than 80 forks as of today. That’s a lot of eyes to catch bugs and improve performance.

But it’s not just a more robust and faster implementation of CSS that works on every browser. It has removed useless CSS selectors and added extra goodies that turn out to be pretty useful for people writing tests.

Sizzle’s extra features you can use right now

As it says in its docs, Sizzle not only implements virtually all CSS 3 Selectors, but also extends them a little bit and adds its own, including some that are actually pretty useful for writing Selenium tests:

:not

The :not selector will help you filter out elements that are similar, but not exactly what you’re looking for. Let’s imagine the following situation:

<a href="#meh" class="confirmation_link hidden">Confirm</a>
... lots of html ...
<a href="#bleh" class="confirmation_link">Confirm</a>

As you can see here, there are two links in the page you’re trying to test. Since they both have the same text, link=Confirm won’t work because they’re the same class and there is no id you could use to be more specific. In this kind of situation, the :not selector is our perfect weapon. It’s just as simple as writing the following locator:

selenium.click("css=a.confirmation_link:not(.hidden)")

Thanks to Sizzle, complex filters can go inside :not. Here are some other examples:

selenium.click("css=.confirmation_link:not(div)")
selenium.click("css=.submit_button:not(#clear_button)")
selenium.click("css=input[type=button]:not(p#not_this_input > input)")

:contains

Even though :contains was already present in cssQuery and the old Selenium, I thought it was worth mentioning. It can be used to filter elements depending on their inner text, so you can do something like:

selenium.click("css=div#myID > a:contains(Confirm Transaction)")

:eq/:nth

This selector finds all the occurrences and then just filters the nth in the list. If are using the confusing :nth-of-type or :nth-child filters, this may be a great replacement.

selenium.click("css=table a:contains(Change password):nth(5)")

:header

This selector will find you any header element. That is, h1, h2, h3, h4, h5 or h6. Pretty cool, huh? This way you can forget about which type of header your devs choose to use in the page. All you care to know is that it’s going to be a header.

Asserting a header that contains a specific text is the perfect situation for this:

assert selenium.is_element_present(":header:contains(Users Admin)")

Form helpers

Additionally, Sizzle includes some form element shortcuts to save you from having to find out whether the element is a textarea element or an input. Even better, it saves you from writing ugly locators like input[type=checkbox].

  • :input: Finds all input elements (includes textareas, selects, and buttons).
  • :text:checkbox:file:password:submit:image:reset:button: Finds the input element with the specified input type (:button also finds button elements).

I think those are all important and you can check out the Sizzle’s wiki for more info. We also released a CSS selectors quick reference if you’d like to have a cheat-sheet printed and close to your desk while you’re writing your tests ;)

Hope everyone is now writing jQuery Selenium selectors and found this post useful for saving some time and headaches. Of course, all of this is already available in our browsers in the cloud service, Sauce OnDemand, so go try it out for free!

#SFSE Meetup: Cloud Testing At Salesforce & Selenium 2 Update

November 29th, 2010 by Ashley Wilson

Here are videos from another great San Francisco Selenium Meetup. Chris Chen, of Salesforce, gave attendees a special look in to how Salesforce automates testing in the cloud, and Eran Messeri, a core Selenium committer at Google, shared a new API for Selenium 2.

Check ’em out!


How Salesforce Tests in the Cloud with Chris Chen


Selenium 2 API & Interactions with Eran Messeri

Parallel Automated Browser Testing in PHP using Selenium and Sauce OnDemand

September 23rd, 2010 by The Sauce Labs Team

Guest post by Aytekin Tank, co-founder of Interlogy Internet Technologies

If you are in SaaS business, one of the best things you can do for yourself is to setup a continuous automated browser testing environment. It will save you a lot of headache, money, and customers down the road.

As a happy new Sauce OnDemand user, I’d like to share our story with automated browser tests and how we integrated a PHP based web app with Sauce OnDemand.

Our web application, JotForm, is a web based form builder. Using its drag and drop WYSIWYG form editor, you can create web forms quickly and easily. Since our core application is written in JavaScript, it is extremely important for us to do automated cross-browser tests. We accomplish this by using Selenium.

We use Hudson to automate our releases. We do continuous integration and deployment. We release our product ten times a day every day. When a developer makes a commit, Hudson executes hundreds of unit, integration, and cross browser tests. Our service has over 300,000 users and they can be very loud when we mess up. That’s why we care a lot about our tests. If we mess up once, we investigate how we missed it and make sure to add new tests so something like that never happens again.

Why Automated Selenium Tests?

If you are developing a web app, you must have already discovered that unit tests don’t work well. They can only really cover the server side and little bit on the JavaScript, but when it comes to testing your code on a multi-browser environment, they are pretty much useless. Browsers constantly change. Just this week, our developers spent a lot of time making our app work properly on IE 9 Beta.

Ideally, you would want to test your app on all browsers and all platforms. Selenium is the best tool for the job. You can easily record tests using Selenium IDE and integrate them with your continuous integration tools, such as Hudson or Cruise Control.

Why NOT Automated Selenium Tests In-house?

The biggest problem with running Selenium tests is creating and managing many testing environments. Usually if you are running Selenium tests constantly on a machine, it cannot be used by anyone. So in practice, they have to be on dedicated machines. And if your releases depend on the Selenium tests, you have to make sure they are always on and running smoothly. It is a lot of maintenance headache.

The other problem with Selenium is that it is slow. It takes Selenium a long time to launch up the browser to do the tests. Unless you have many dedicated machines, the tests have to run sequentially.

To speed up our Selenium tests, I looked for ways to run them in parallel. My first approach was to use Amazon EC2, since I am already pretty familiar with it. That turned out to not be a great solution. We still had to maintain many selenium environments. Keeping many EC2 instances always on can be quite expensive. Launching new instances on demand turned out to be too slow for our case.

Final Solution: Sauce OnDemand

Then, I found out about Sauce Labs’ Sauce OnDemand service. This is basically an on demand service that lets you run Selenium tests instantly on the cloud. It has many good code samples. So, setting it up was pretty straightforward. The great thing about Sauce OnDemand is that you can choose from many browser setups and run your tests on them instantly. No need to maintain dedicated machines.

Unfortunately, there was no documentation about running tests in parallel in PHP. So, we first looked into using PHPUnit’s new parallel processing features: @runTestsInSeparateProcesses and @runInSeparateProcess. They turned out to be too buggy to be usable at this time. So, we hacked together a simple script that runs the tests on the shell as background processes.

A Simple Example

I am a big Perl junkie, but since our existing unit and integration tests were written in PHPUnit, we wanted to keep our Selenium tests consistent and write them in PHP. Below, you can find a simple example showing how you can run automated cross-browser Sauce OnDemand tests on PHP. We hacked this together quickly so it is not documented or optimized. But it should give you a good jump-start:

1. BrowserTests.php: Includes actual Selenium tests and simplified for this example with a single test. On our production version, we have all the tests here, since we prefer to save time and run all tests at once on a specific browser. You can create these tests using Selenium IDE or Sauce IDE.

2. allBrowserTests.php: You can run this on the shell then add it to Hudson or any other Continuous Integration tool. Basically, this script runs Selenium tests on the background in parallel. It then checks their log files to see if they finished successfully or failed.

3. SeleniumTestHelper.php: This is the integration script with Sauce Labs. You should change the domain name, Sauce username and access key to your own.

Download all files files here. You must first have phpunit installed to use them.

Happy testing!

Aytekin Tank is a co-founder of Interlogy Internet Technologies. Their flagship product is JotForm online form builder.

RSpec + Selenium Demonstration 9/30 at 11 am PST

September 21st, 2010 by Ashley Wilson

Join Sean Grove, senior developer at Sauce Labs, as he shows you how to take your RSpec tests and pit ’em against the real world. RSpec is a Ruby framework that provides a Domain Specific Language that expresses executable examples of the expected behavior of your code.

Learn how to launch and control real browsers directly from RSpec tests using Selenium and Sauce OnDemand. In this live video demontration on Sept. 30 at 11 am PST, he’ll demonstrate and discuss:

* Writing your first RSpec test
* Configuring it to use Selenium
* Cross-browser testing and spec parallelization

Be sure to stick around for the Q&A following the presentation. Register Here!

Selenium Tips: How to Coordinate Multiple Browsers in Sauce OnDemand

September 3rd, 2010 by Santiago Suarez Ordoñez

Let’s imagine your team is making a chat application. Nothing too fancy, just something simple in which you need to make sure multiple users get together, talk, and read each other.

Doesn’t sound that bad, huh? But how about coordinating those browsers? You need to have different sessions running at the same time in coordination and interacting with each other through your web app.

Now, when we think about parallelization and multiple browsers, Sauce OnDemand is, of course, our first answer, but here, there can be a small limitation. Sauce OnDemand has a special timeout that waits for new commands for no more than 90 seconds*. If it doesn’t get any new command, it will kill the job and assume something went wrong.

This effects us, as we need to get multiple browsers running in parallel and in coordination. As we request them one after another, depending on the amount of browsers and the load our service is put under, the first ones might timeout while waiting for the others to come.

To avoid this, the best solution is to write a multi-threaded script, in which browsers will send heartbeat commands while they are waiting for the rest. Multi-threading can be done with every programming language, but I decided to write my example in Python because I think it’s understandable for almost anyone, even if you’ve never seen Python before. The complete example is in github as a gist, and here is a brief description of what each interesting part is doing:

def get_browser_and_wait(browser, browser_num):
    print "starting browser %s" % browser_num
    browser.start()
    browser.open("/") # if we get here, OnDemand already gave us the browser
    browsers_waiting.append(browser)
    print "browser %s ready" % browser_num
    while len(browsers_waiting) < len(browsers):
        print "browser %s sending heartbeat while waiting" % browser_num
        browser.open("/")
        time.sleep(3)

get_browser_and_wait is the function that will take care of the ‘start the browser’ request in OnDemand and wait till it comes back. Once the server is ready, it will keep sending open commands as heartbeats and waiting until the other requested browsers are ready and waiting too.

thread_list = []
for i, browser in enumerate(browsers):
    t = Thread(target=get_browser_and_wait, args=[browser, i + 1])
    thread_list.append(t)
    t.start()

for t in thread_list:
    t.join()

This is the magic multithreading part, in which we iterate over every Selenium instance and call get_browser_and_wait with it. Once we send each browser to a thread, the main thread will continue and get to the t.join() part. By this method, the main thread will wait for every browser thread to complete in order to proceed with the rest of the code.

Again, the full example is up in github: http://gist.github.com/511658. If you’re interested in learning what you can use this kind of test for, stay tuned!

* This helps avoid running (and charging) tests that get disconnected or crash.

Trends in Testing: Continuous Integration, Mobility, Open Source, Cloud

May 10th, 2010 by The Sauce Labs Team

Functional testing is the automation of web app testing across several platforms or browsers. In the past, functional testing was less common due to long development cycles and the lack of multiple browser options. It was done primarily in-house via proprietary software like Rational or QuickTestPro.

Today, there are four trends converging to radically change how organizations test web applications:

  1. Continuous integration – An increased emphasis on performance of the software through the end customer’s perspective is leading agile development. Gartner predicts that by 2012 agile development methodologies will be used by 80 percent of all software development projects. Teams are shifting away from rigid quality control to quality that is demonstrable to the end user. With this shift, comes a need for quick, simple and automated testing tools.
  2. Open source – Open source tools, like Selenium, are viable for functional testing. Today’s development environment has dramatically changed and a lot more people are contributing than ever before. The major advantages of open-source are speed, time to market, time to value, and the ability to reach and grow a developer and user community. In fact, Gartner predicts that by 2012, 80 percent of all commercial software will include elements of open source technology.
  3. Cloud computing – Testing in the cloud is an affordable and scalable alternative to testing behind a firewall. It is expensive and time consuming to maintain on site test infrastructures that cover a vast number of browsers (and versions) of operating systems in several languages. The increase in the adoption of cloud computing creates an opportunity to leverage the space for functional testing. We believe the life cycle for cloud-based applications will look different in the future. New solutions that are cloud based will support cloud-base applications.
  4. Mobility – Websites need to support multiple browsers such as Internet Explorer, Firefox, Chrome, Rockmelt, Opera on various platforms including Windows and Mac. Smartphones only add to this “Browser War 2.0” battle with the plethora of mobile operating systems. These various environments make the infrastructure required more complex and the need for functional testing more relevant that ever before.

Are there other trends beyond continuous integration, mobility, open source and the cloud that will change how organizations test their web applications?

Are Desktop Applications Dying?

May 4th, 2010 by Jason Huggins

ZDNet Blogger John Carrol posted a blog last week titled “Everybody hates browser applications” that has started a lively discussion. I feel that “everybody” certainly does not hate browser applications. In fact, I feel just the opposite. That is, despite the resistance, desktop applications are dying.

Browser applications are ‘good enough’ compared to their desktop equivalents and are getting better all the time. And even then, it doesn’t matter if everyone hates browser applications, they’re going to be using browser applications anyway because they’re easier for developers to write and distribute. Everyone’s a critic, but it’s the developers who dictate which technology gets used to create apps.

Example #1 how browser applications are ‘good enough’ compared to their desktop equivalents

At the recent JSConf I attended, I saw a demo of the new iPad support in SproutCore (the JavaScript framework that Apple uses for its MobileMe online web applications.) The iPad team worked with the SproutCore team to provide native hardware acceleration to the JavaScript layer and made the web applications just as fast and native looking as their native siblings written in Objective-C.

Example #2 how browser applications are ‘good enough’ compared to their desktop equivalents

One of the more famous iPad apps is the NPR app, a native Objective-C iPad that launched at the same time as the iPad. At JSConf, a JavaScript-only SproutCore version of the NPR app was written in a few days, and actually was faster than the native equivalent. The JavaScript/web version was ‘better’ than the native version.”

So consider this – as a web-developer looking to get into the mobile marketplace, wouldn’t you rather leverage your existing skill set (HTML, CSS, and JavaScript) than learn a completely new technology set (Objective-C for iPad and iPhone)?

Sauce OnDemand supports Firefox 3.6

February 3rd, 2010 by The Sauce Labs Team

Following our zero-day support for Firefox 3.6 in Sauce RC, we are pleased to announce that Sauce OnDemand now supports Firefox 3.6 as well.

Use the following settings in your JSON configuration:

{
    "username" : "SAUCE-USER-NAME",
    "access-key" : "SAUCE-API-KEY",
    "os" : "Windows 2003",
    "browser" : "firefox",
    "browser-version" : "3.6."
}