Posts Tagged ‘cucumber’

Write Great Cucumber Tests

March 14th, 2016 by Greg Sypolt

Using Cucumber with outlined best practices in your automated tests ensures that your automation experience will be successful and that you’ll get the maximum return on investment (ROI). Let’s review some important best practices needed before you start developing Cucumber tests.

Writing Features

Feature files help non-technical stakeholders engage and understand testing, resulting in collaboration and bridging the communication gap. For this reason, well-written feature files can be a valuable tool for ensuring teams use the BDD process effectively.

Here are some suggested standards for Cucumber Feature files:

OrganizationFeature files can live at the root of the /features directory. However, features can be grouped in a subfolder if they describe a common object. The grouped filenames should represent the action and context that is covered within.

Feature FilesEvery *.feature file consists in a single feature, focused on the business value.
Gherkin TemplateFeature: Title (one line describing the story)

Narrative Description: As a [role], I want [feature], so that I [benefit]

Scenario: Title (acceptance criteria of user story)
Given [context]
And [some more context]...
When [event]
Then [outcome]
And [another outcome]...

Scenario: ...
BackgroundThe background needs to be used wisely. If you use the same steps at the beginning of all scenarios of a feature, put them into the feature’s background scenario. The background steps are run before each scenario.
ScenariosKeep each scenario independent. The scenarios should run independently, without any dependencies on other scenarios.
Scenario OutlineIf you identify the need to use a scenario outline, take a step back and ask the following question: Is it necessary to repeat this scenario ‘x’ amount of times just to exercise the different combination of data? In most cases, one time is enough for UI level testing.
Write Declarative Scenarios, Not ImperativeThe declarative style describes behavior at a higher level, which I think improves the readability of the feature by abstracting out the implementation details of the application.

Example: Imperative

Scenario: User logs in
Given I am on the homepage
When I click on the "Login" button
And I fill in the "Email" field with "me@example.com"
And I fill in the "Password" field with "secret"
And I click on "Submit"
Then I should see "Welcome to the app, John Doe"


Example: Declarative

Scenario: User logs in
Given I am on the homepage
When I log in
Then I should see a login notification


Just avoid unnecessary details in your scenarios.
Given, When, and Then StatementsI’ve often seen people writing the Gherkin syntax confuse when to put the verification step in the Given, When, Then sequence. Each statement has a purpose.
  • Given is the pre-condition to put the system into a known state before the user starts interacting with the application

  • When describes the key action the user performs

  • Then is observing the expected outcome

Just remember the ‘then’ step is an acceptance criteria of the story.
TaggingSince tags on features are inherited by scenarios, please don’t be redundant by including the same tags on scenarios.
(more…)

Guest Post: Test Lessons at ExpoQA

June 6th, 2014 by Bill McGee

This is the first of a three part series by Matthew Heusser, software delivery consultant and writer. 

Every now and again and opportunity comes along that you just can’t refuse. Mine was to teach the one-day version of my class, lean software testing, in Madrid, Spain, then again the following week in Estonia. Instead of coming back to the United States, I’ll be staying in Europe, with a few days in Scotland and a TestRetreat in the Netherlands.

And a lot of time on airplanes.

The folks at Sauce Labs thought I might like to take notes and type a little on the plane, to share my stories with you.

The first major hit in Madrid is the culture shock; this was my first conference where English was not the primary language. The sessions were split between English and Spanish, with translators in a booth making sure all talks were available in all languages.

The Testing Divide

Right now, in testing, I am interested in two major categories: The day to day work of testing new features and also the work of release-testing after code complete. I call this release testing a ‘cadence’, and, across the board, I see companies trying to compress the cadence.

My second major surprise in Madrid is how wide the gap is —and I believe it is getting wider —between legacy teams that have not modernized and teams starting from scratch today. One tester reported a four-month cycle for testing. Another team, relying heavily on Cucumber and Selenium, were able to release every day.

Of course, things weren’t that simple. The Lithuanian team used a variety of techniques I can talk about in another post to reduce risk, something like devOps, which I can talk about in another post. The point here is the divide between the two worlds.

Large cadences slow down delivery. They slow it down a lot; think of the difference between machine farming in the early 20th century and the plow and horse of the 19th.

In farming, the Amish managed to survive by maintaining a simple life, with no cars, car insurance, gasoline, or even electricity to pay for. In software, organizations that have a decades-long head start: banks, insurance companies, and pension funds, may be able to survive without modernization.

I just can’t imagine it will be much fun.

Batches, Queues and Throughput

Like many other conferences, the first day of ExpoQA is tutorial day, and I taught the one-day version of my course on lean software testing. I expected to learn a little about course delivery, but not a lot —so the learning hit me like a ton a bricks.

The course covers the seven wastes of ‘lean’, along with methods to improve the flow of the team – for example, decreasing the size of the measured work, or ‘batch size’. Agile software development gets us this for free, moving from ‘projects’ to sprints, and within sprints, stories.

In the early afternoon we use dice and cards to simulate a software team that has equally weighted capacity between analysis, dev, test and operations —but high variability in work size. This slows down delivery. The fix is to reduce the variation, but it is not part of the project, so what the teams tend to do is to build up queues of work, so any role never runs out of work.

What this actually does is run up the work in progress inventory – the amount of work sitting around, waiting to be done. In the simulation I don’t penalize teams for this, but on real software projects, ‘holding’ work created multitasking, handoffs, and restarts, all of which slow down delivery.

My lesson: Things that are invisible look free —and my simulation is far from perfect.

After my tutorial it is time for a conference day – kicked off by Dr. Stuart Reid, presenting on the new ISO standard for software testing. Looking at the schedule, I see a familiar name; Mais Tawfik, who I met at WOPR20.Mais is an independent performance consultant; today she is presenting on “shades of performance testing.”

Performance Test Types

Starting with the idea that performance testing has three main measurements: Speed, Scalability, and Stability, Mais explains that there are different types of performance tests, from front-end performance (javascript, waterfalls of HTTP requests, page loading and rendering) to back-end (database, webserver), and also synthetic monitoring – creating known-value transactions continuously in production to see how long they take. She also talks about application usage patterns – how testing is tailored to the type of user, and how each new release might have new and different risks based on changes introduced. That means you might tailor the performance testing to the release.

At the end of her talk, Mais lists several scenarios and asks the audience what type of performance test would blend efficiency and effectiveness. For example, if a release is entirely database changes, and time is constrained, you might not execute your full performance testing suite/scripts, but instead focus on rerunning and timing the database performance. If the focus on changes is the front end, you might focus on how long it takes the user interface to load and display.

When Mais asks if people in the organization do performance testing or manage it, only a handful of people raise their hands. When she asks who has heard of FireBug, even less raise their hand.

Which makes me wonder if the audience is only doing functional testing. If they are, who does the performance testing? And do they not automate, or do they all use Internet Explorer?

The talk is translated; it is possible that more people know these tools, it was just that the translator was ‘behind’ and they did not know to raise their hands in time.

Here’s hoping!

Time For A Panel

At the end of the day I am invited to sit on a panel to discuss the present (and future) of testing, with Dr. Reid, Dorothy Graham, Derk-Jan De Grood, Celestina Bianco and Delores Ornia. The questions include, in no particular order:

  •             Will testers have to learn to code?
  •             How do we convince management of the important of QA and get included in projects?
  •             What is the future of testing? Will testers be out of a job?
  •             What can we do about the dearth of testing education in the world today?

For the problem with the lack of education, Dorothy Graham points to Dr. Reid and his standards effort as a possible input for university education.

When it is my turn, I bring up ISTQB The International Software Testing Qualifications Board. – if ISTQB is so successful (“300,000 testers can’t be wrong?”) then why is the last question relevant? Stefaan Luckermans, the moderator, replied that with 2.9 Million testers in the world, the certification had only reached 10%, and that’s fair, I suppose. Still, I’m not excited about the quality of testers that ISTQB turns out.

The thing I did not get to say, because of time, that I want to do is point out that ISTQB is, after all, just a response to a market demand for a 2-3 day training certification. What can a trainer really do in 2-3 days? At most, maybe, teach a single technical tool, turn the lightbulb of thinking on, or define a few terms. ISTQB defines a few terms, and it takes a few days.

The pursuit of excellent testing?

That’s the game of a lifetime.

By Matthew Heusser – matt.heusser@gmail.com for Sauce Labs

Stay tuned next week for part two of this mini series! You can follow Matt on Twitter at @mheusser.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Setting up iOS Automation on Sauce Labs with Cucumber and Appium

October 7th, 2013 by Shashikant Jagtap

Automated mobile testing with Cucumber & AppiumAbstract

Sauce Labs has recently announced Appium support which makes it easier to test mobile apps in the cloud. Appium is a mobile test automation framework for hybrid and native mobile apps. Cucumber is a behaviour driven development a.k.a BDD tool used with different programming languages. The combination of Cucumber and Appium can be used for automating iOS apps in the cloud using Sauce Labs. This is a repost of the original post. In this post, we will see how to set up test automation of our iOS app in the cloud using Sauce Labs.

Getting Started

In order to get started, we need to have the initial setup handy. This includes the following:

  • Mac OSX 10.7.4 or higher with Xcode installed with command line tools.
  • Your app source code or a prebuilt .app bundle for your app. Browse wide range of open-source iOS apps
  • Saucelabs Username and API key. Signup for Saucelabs free account.
  • Web development environment on Mac OSX for Ruby including Xcode, RVM, HomeBrew, Git and Ruby. Follow Moncef’s blog
  • Appium-Ruby-console  with Node and npm (Node must be >= v0.8)
  • Ruby 1.9.3

Get Appium and Your iOS App

Before we go ahead, let’s get the Appium server up and running. There are two ways to do it-

  •  You can download Mac OSX Appium.dmg package and launch Appium.
  • You can run it from source. Follow instructions. You need to make sure, You have authorized use of the iOS Simulator. If you are running Appium from NPM, you’ll do this by running
$ sudo authorize_ios

There are a wide range of open source iOS mobile apps available here, we are going to use PlainNote iOS app for this tutorial.

Compile and Upload iOS App on SauceLabs

Now we need to compile PlainNote App with Sauce. [ Note Additional parameter TARGET_DEVICE_FAMILY]

$ git clone https://github.com/vkoser/PlainNote
$ cd PlainNote
$ xcodebuild -sdk iphonesimulator6.0 TARGETED_DEVICE_FAMILY=1

In Sauce, there are optional parameters like TARGETED_DEVICE_FAMILY parameter . To build an app for iPhone, we use  TARGETED_DEVICE_FAMILY= 1, for iPad  TARGETED_DEVICE_FAMILY=2 .

Now, once the build is successful, it will create “PlaneNote.app” at ‘/build/Release-iphonesimulator‘.

$ cd /build/Release-iphonesimulator/
$ ls
PlainNote.app PlainNote.app.dSYM
  •  Zip the PlainNote App & Upload to SauceLabs 

Now that we have ‘PlainNote.app‘, we need to zip it by navigating to that directory

$ cd /build/Release-iphonesimulator/
$ zip -r PlainNote.zip PlainNote.app

Now you will see ‘PlainNote.zip‘ file in that directory

$ ls
PlainNote.app      PlainNote.app.dSYM PlainNote.zip

Now, we need to upload this file to Sauce Labs temporary storage using the Sauce REST API. I am using my Username and API key here.

$ curl -u Shashikant86:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxbfc3 -X POST "http://saucelabs.com/rest/v1/storage/Shashikant86/PlainNote.zip?overwrite=true" -H "Content-Type: application/octet-stream" --data-binary @/path/to/PlainNote/build/Release-iphonesimulator/PlainNote.zip

It will show you response something like this,

{"username": "Shashikant86", "size": 42190, "md5": "6ef42125b024188976af9d6b8a104105", "filename": "PlainNote.zip"}

It will be now uploaded to “sauce-storage:PlainNote.zip“. Now we are all set for writing tests for the application with Cucumber.

Setup Cucumber project

Now that we have already uploaded our app on SauceLabs temporary storage, we can setup Cucumber project to talk to the mobile app in the cloud. I assume that you are familiar with the BDD Code and code structure for the Cucumber project. Please refer my old post if you are not aware of BDD code.

  • Create Gemfile

We need a Gemfile in order to specify all our dependencies

$ mkdir sauce-cucumber-appium
$ cd sauce-cucumber-appium
$ rvm use 1.9.3
$ vim Gemfile

Now insert the following dependencies into the Gemfile

source "https://www.rubygems.org"
gem "rest-client"
gem "rspec"
gem "selenium-webdriver"
gem "cucumber"
gem "rspec-expectations"

Now we need to install the bundle to download all the dependencies.

$ bundle install

This will create a ‘Gemfile.lock’ file.

  •  Create Feature File 

Now, we will write a feature file using Given When Then format. The feature file is written using the Gherkin Domain Specific Language.

Let’s create ‘features/plain_note_sauce.feature‘ file.

$ vim features/plain_note_sauce.feature

The PlainNote app feature will look something like this

Feature: Notes 
  As iOS automation specialist 
  I want to setup iOS app automation in the cloud using Saucelabs, Appium and cucumber 

Scenario: Add new Note using PlainNote App 

  Given I have App running with appium on Sauce 
  When click + button using sauce driver
  And I enter text "Data" and saved it on sauce
  Then I should see "Data" note added on home page in the sauce cloud

This feature is self explanatory, we are going to add a new note and make sure it displayed on the Home page.

  • Setup Cucumber Environment  

Let’s create ‘features/support/env.rb‘ where we can put our support code. We need to add sauce_capabilities mentioned in the Sauce Labs Appium tutorial.

$ vim features/support/env.rb

Insert the following code in the file.

require 'rspec/expectations'
require 'selenium-webdriver'

def sauce_capabilities
  {
    'app' => 'sauce-storage:PlainNote.zip',
    'device' => 'iPhone Simulator',
    'username' => 'Shashikant86',
    'access-key' => 'a0e37e25-e2f3-4cba-95d3-936007d8bfc3',
    'platform' => 'OS X 10.8',
    'version' => '6.0',
    'name' => 'Running PlainNote wit Cucumber and Appium', 
    'passed' => 'true'
  }
end

def sauce_url
  "http://Shashikant86:a0e37e25-e2f3-4cba-95d3-936007d8bfc3@ondemand.saucelabs.com:80/wd/hub"
end

def sauce
  @sauce ||= Selenium::WebDriver.for(:remote, :desired_capabilities => sauce_capabilities, :url => sauce_url)
end

After { @sauce.quit }

Now that we have  a created ‘sauce’ driver with all required desired capabilities, ee will using the ‘sauce’ object in our step_definitions

  •  Write Step definitions using Selenium-Webdriver  JSON Wire Protocol

At this point if you run the ‘bundle exec cucumber’ command it will tell you steps that are not implemented yet. We need to implement these step definitions using Selenium-Webdriver JSON Wire Protocol for Appium. Now we will create a step definition file and implement it

$ vim features/step_definitions/plain_note.rb

Now add these step definitions to the file.

Given(/^I have App running with appium on Sauce$/) do
end

When(/^click \+ button using sauce driver$/) do
sauce.find_element(:name, "Add").click
end

When(/^I enter text "(.*?)" and saved it on sauce$/) do |data|
sauce.find_element(:xpath, "//window[1]/scrollview[1]/textview[1]").send_keys data
sauce.find_element(:name, "Done").click
sauce.find_element(:name, "Save").click
end

Then(/^I should see "(.*?)" note added on home page in the sauce cloud$/) do |text|
note = sauce.find_element(:xpath,  "//window[1]/tableview[1]/cell[1]/text[1]")
note.attribute("value").should match text 
end

Appium Inspector 

Appium Inspector is a feature of the Appium OSX app which allows you to inspect elements on your mobile app. You can also record tests in the different languages. Writing the Ruby code is easy if you have used Appium Inspector locally to record tests. Watch this video to know ‘How to use Appium Inspector‘.

Now, we are all set to run cucumber to execute tests on the SauceLabs

$ bundle exec cucumber features/plain_note_sauce.feature

Now you will see the tests running on Sauce Labs and in your terminal you will see something like this

Feature: Notes
  As iOS automation specialist
  I want to setup iOS app automation in the cloud using Saucelabs, Appium and cucumber

  Scenario: Add new Note using PlainNote App                            # features/plain_note_sauce.feature:5
    Given I have App running with appium on Sauce                       # features/step_definitions/plain_note_sauce.rb:1
    When click + button using sauce driver                              # features/step_definitions/plain_note_sauce.rb:4
    And I enter text "Data" and saved it on sauce                       # features/step_definitions/plain_note_sauce.rb:8
    Then I should see "Data" note added on home page in the sauce cloud # features/step_definitions/plain_note_sauce.rb:14

1 scenario (1 passed)
4 steps (4 passed)
0m35.213s

You can watch video and screenshots of the job.

Cucumber-Appium-SauceLabs

You can find source code on GitHub:

GitHub : cucumber-appium

Watch this video get a clear idea of the cucumber-appium setup on Sauce Labs. If any questions, feel free to contact me.

#SeConf Videos Now Available!

May 2nd, 2011 by Ashley Wilson

In case you missed the awesome Selenium Conference that happened in early April, check out videos from each of the presentations. We’ll be posting a couple at a time, so stay tuned to future posts!


Jason Huggins’ Opening Keynote


Dave Hunt & Andrew Smith: Automating Canvas Applications
Despite recent improvements to automated testing tools, there’s still a large gap when it comes to emerging technologies such as HTML5. Recent developments like the canvas element present an interesting dilemma for traditional automated testing as they expose little or no information to debug tools. In order to move forwards, both developers and testers will need to work together. Using Selenium, Java, and JavaScript we will demonstrate writing automated tests for a popular canvas game.


Dima Kovalenko: Selenium and Cucumber
Wouldn’t it be nice to have the BA’s write out the acceptance criteria in plain English, and then have those criteria run as tests? Join us for a beginner to intermediate walk through of Cucumber and Selenium. Learn how to write tests that are easy to understand and run. There will be plenty of examples and sample code to get you going in the right direction.

Cucumber Sauce: Cross-browser testing in parallel

September 19th, 2010 by The Sauce Labs Team

I’ve been meaning to release a “standard” cucumber-in-parallel cross-browser project for awhile now, but I just got around to it this weekend. It’s called Cucumber Sauce.

Highlights:

  • Run your cucumber tests across different browsers all at the same time
  • Browsers are configurable via a yaml file
  • Multiple browser config files are easy to setup, and one can be passed in when running the rake task. Have a browsers_core.yml file to run while developing, and a browsers_full.yml to run before pushing to production
  • Designed to run tests ultra-cleanly. Adds time overhead initially, but tests are all independent of one another, and can be made to run in parallel at the scenario level in the future. Follow this template or you’ll lose that ability to speed up your tests.

Check it out at the Cucumber Sauce github repo.

Setting up Cucumber + Webrat + Selenium

June 15th, 2010 by The Sauce Labs Team

There’s quite a bit of information out there on getting these disparate tools to work together, but a great deal of it is out of date. To clear things up a bit, I’ve documented all the gems and modifications necessary to get these pieces of open source software up and running together. As time goes on, I’ll be expanding this blog post with notes about the pitfalls and various platform issues that may be discovered (I’m looking at you, Snow Leopard), but this should get most people up and running right away.

This is all using a clean REE environment via the poorly named but wonderfully written rvm, or Ruby Version Manager.

Gems you’ll need:

gem install actionmailer actionpack activerecord activeresource activesupport builder cgi_multipart_eof_fix cucumber cucumber-rails daemons database_cleaner diff-lcs fastthread gem_plugin gherkin json json_pure mime-types mongrel net-ssh net-ssh-gateway nokogiri rack rack-test rails rake rdoc rest-client rspec rspec-rails Selenium selenium-client sqlite3-ruby term-ansicolor trollop webrat

Some of those are not strictly necessary, but simply nice to have, while others solved some unexpected problems with the bare necessities. I’ll prune this list as feedback comes in from people’s experiences.

Points to watch out for:
Nokogiri: This was easily the worst on my Snow Leopard machine. It relies on the native libxml2, which had problems with 32/64 bit compatibility. No matter what I tried, errors kept coming up. I had to clean everything out with my MacPorts installation and force a universal installation.
Webrat: The Selenium server jar that was included by default caused no end of headaches. I had to manually go in to the directory, remove the default server jar, and download the newest version from the seleniumhq download page.

Once those are set up, you should be able to use Cucumber, Webrat, and Selenium together without too much headache. Then you can refer to our webinar video (which will be posted to the blog later this week) to learn how to use Cucumber to easily run Webrat and Selenium.