Guest post by Aytekin Tank, co-founder of Interlogy Internet Technologies
If you are in SaaS business, one of the best things you can do for yourself is to setup a continuous automated browser testing environment. It will save you a lot of headache, money, and customers down the road.
As a happy new Sauce OnDemand user, I’d like to share our story with automated browser tests and how we integrated a PHP based web app with Sauce OnDemand.
We use Hudson to automate our releases. We do continuous integration and deployment. We release our product ten times a day every day. When a developer makes a commit, Hudson executes hundreds of unit, integration, and cross browser tests. Our service has over 300,000 users and they can be very loud when we mess up. That’s why we care a lot about our tests. If we mess up once, we investigate how we missed it and make sure to add new tests so something like that never happens again.
Why Automated Selenium Tests?
Ideally, you would want to test your app on all browsers and all platforms. Selenium is the best tool for the job. You can easily record tests using Selenium IDE and integrate them with your continuous integration tools, such as Hudson or Cruise Control.
Why NOT Automated Selenium Tests In-house?
The biggest problem with running Selenium tests is creating and managing many testing environments. Usually if you are running Selenium tests constantly on a machine, it cannot be used by anyone. So in practice, they have to be on dedicated machines. And if your releases depend on the Selenium tests, you have to make sure they are always on and running smoothly. It is a lot of maintenance headache.
The other problem with Selenium is that it is slow. It takes Selenium a long time to launch up the browser to do the tests. Unless you have many dedicated machines, the tests have to run sequentially.
To speed up our Selenium tests, I looked for ways to run them in parallel. My first approach was to use Amazon EC2, since I am already pretty familiar with it. That turned out to not be a great solution. We still had to maintain many selenium environments. Keeping many EC2 instances always on can be quite expensive. Launching new instances on demand turned out to be too slow for our case.
Final Solution: Sauce OnDemand
Then, I found out about Sauce Labs’ Sauce OnDemand service. This is basically an on demand service that lets you run Selenium tests instantly on the cloud. It has many good code samples. So, setting it up was pretty straightforward. The great thing about Sauce OnDemand is that you can choose from many browser setups and run your tests on them instantly. No need to maintain dedicated machines.
Unfortunately, there was no documentation about running tests in parallel in PHP. So, we first looked into using PHPUnit’s new parallel processing features: @runTestsInSeparateProcesses and @runInSeparateProcess. They turned out to be too buggy to be usable at this time. So, we hacked together a simple script that runs the tests on the shell as background processes.
A Simple Example
I am a big Perl junkie, but since our existing unit and integration tests were written in PHPUnit, we wanted to keep our Selenium tests consistent and write them in PHP. Below, you can find a simple example showing how you can run automated cross-browser Sauce OnDemand tests on PHP. We hacked this together quickly so it is not documented or optimized. But it should give you a good jump-start:
1. BrowserTests.php: Includes actual Selenium tests and simplified for this example with a single test. On our production version, we have all the tests here, since we prefer to save time and run all tests at once on a specific browser. You can create these tests using Selenium IDE or Sauce IDE.
2. allBrowserTests.php: You can run this on the shell then add it to Hudson or any other Continuous Integration tool. Basically, this script runs Selenium tests on the background in parallel. It then checks their log files to see if they finished successfully or failed.
3. SeleniumTestHelper.php: This is the integration script with Sauce Labs. You should change the domain name, Sauce username and access key to your own.
Download all files files here. You must first have phpunit installed to use them.
Aytekin Tank is a co-founder of Interlogy Internet Technologies. Their flagship product is JotForm online form builder.