Translating Web App Functional Testing To Mobile

April 27th, 2015 by Amber Kaplan

This is a guest post by Greg Sypolt, a Senior Engineer at Gannett Digital and automated testing expert.

Technological advancements and the explosion of devices across multiple platforms means hardware and software developers have a much more difficult job. They have to keep up with the demand to develop and roll out new products. One of the most significant issues is accounting for the differences in system response, when responding to mobile traffic rather than to internet traffic.

Applications must be tested to make certain they run responsively on key platforms and across numerous networks. Effective functional testing eases the pressure on device manufacturers while allowing application developers the time to collect applicable metrics that improve product quality.

A New Set of Challenges

Testing mobile applications is completely different and significantly more complicated than testing traditional desktop and net applications. Mobile devices don’t just emulate the desktop environment — they have their own set of requirements. Mobile app testing is far trickier because of the following key aspects.

Device Variation:

Mobile applications run on devices that have different:

  • Operating Systems (OS) such as iOS, Android, Windows and BB;
  • Versions of those OS; and
  • Manufacturers such as Apple, Samsung, Nokia, Motorola and LG.

When an application needs to run on multiple OS, devices, and varied screen sizes, QA teams are faced with the challenge of ensuring the application functions in every environment. The below graphic of the iOS support matrix alone shows the complexity with a single series of devices and OS type.

source: iossupportmatrix.com

source: iossupportmatrix.com

Availability of Mobile Testing Tools:

Desktop and web application-based testing tools cannot be used for testing mobile applications, so a new testing framework is required. Some of the frameworks currently available for writing test scripts are:

Translating 2

Network Challenges:

You must be able to effectively emulate various bandwidth rates, because your end users will be operating on a variety of networks and bandwidths. You also must be able to test geographically isolated loads accurately to signify real-world traffic.

Screen Size and Densities:

With the diversity of screen sizes and densities available today, you must also be able to test your mobile app on different screen configurations so that it:

  • Fits on small screens;
  • Takes advantage of the additional space on larger screens while still looking good on smaller screens. For example, the difference in screen size between iPhone 4 and iPhone 6+;
  • Is optimized for both landscape and portrait orientations.

Peripheral Mode:

When testing in mobile, there are physical elements that also need to be considered, unlike with web applications. You may need to consider wireless and wired peripheral mode testing.

  • Wireless to the device such as near field communication (NFC), Bluetooth or stylus; and
  • Wired internal to the device such as a headphone jack or external testing with cc reader or bar code scanner;
  • Native Applications and Browser-Based Applications.

These are the variables that come with a mix of native and browser-based applications. Native applications reside on the user’s device and communicate over HTTP(s). The browser-based application uses a modified version of a browser to access applications online. As many companies use both applications to offer solutions to their customers, you must support testing of all types of mobile applications.

Taking Challenges on “Strategy First”

What do these testing challenges mean for web developers and site owners? Primarily, for every web application designed, you must also address a strategy to test the product in the mobile space. Building an app with all the features and functionality needed by the client and user is important. Having a rigorous mobile testing plan in place before the mobile app is deployed is even more crucial to its success.

Mobile applications are becoming more and more sophisticated, significantly increasing the requirement for functional testing. To tackle this, organizations that require app testing are always exploring alternatives to traditional manual testing.

Mobile automation testing is a highly effective approach to mobile app QA. It provides significant business returns when executed using the right tools and infrastructure, while factoring in cross-platform challenges. From sites to web applications to native mobile applications, test automation tools bring full-featured functional testing to mobile platforms.

Strategy for Mobile Test Automation

Automated testing can sometimes be a black hole. The best thing to do is automate something and measure that against a precise objective that can be measured in a realistic timeframe. Knowing your hard or soft Return on Investment (ROI) goals helps as well.

Translating 3

Mobile testing requires a balanced approach, and the key is understanding your company’s mobile strategy.

Target Device Selection:

You simply cannot test applications on every device that exists. However, cloud-based device emulation tools allow you to increase that breadth more and more. The best approach is analyzing the market and choosing a representative device that reduces the effort of executing multiple test cases. A few factors to consider are OS version, screen resolution and form factor (smartphone and tablets) while also ensuring the multi-device and multi-platform compatibility of the app.

Emulators:

Emulators mimic the software and hardware environments found on actual devices and provide excellent options. Options like the ability to bypass the network and use a real-world environment via modem where actual users run and interact with those applications on their devices. At the same time you will need to test on a physical device, but this can usually be done more ad-hoc for most applications. Find the right mix of emulators and physical devices to provide the best results!

Tool Selection Criteria:

Test automation tools create a framework to systemize the testing of mobile native and web apps across platforms. For instance, Appium is one of the few frameworks that can develop cross-platform scripts in multiple scripting languages. Collaboration between iOS and Android developers is crucial to ensuring every element that looks the same has the same accessibility label and builds a testable app.

An important step in this process is creating a list of requirements to review when choosing a tool for evaluation. Some questions to ask as you determine your requirements are:

  • Am I looking for a Behavior Driven Development (BDD) framework?
  • Do I need to support native, hybrid, mobile first (responsive web design) or mobile web?
  • Do I need to support diverse mobile platforms such as Android, iOS or Windows?
  • Am I looking to test locally or in the cloud to reduce the cost of ownership?
  • Am I looking to test against emulators or real devices?
  • Do I need a framework that supports cross-platform scripting?
  • Do I need a framework that offers an easy interface for tests to auto-generate scripts? For instance, scripts can be automated without any programming or scripting language knowledge.
  • Am I practicing continuous integration and need a tool that integrates into the broader environment seamlessly?
  • Am I looking for a framework that will attract existing developers to contribute to my automation goals?

Take a step back and consider your resources. Does your QA team have sufficient programming knowledge for automation development? For automation, you must have people with some programming knowledge. If not, do they have the technical capabilities to easily adapt to the new technologies?

Answers to these questions will help guide your team to picking the best framework, and also understand the capabilities of the team to execute on testing.

Automation Environment:

Automation environment and setup depends on the approach to testing. Automation testing approaches are either cloud-based or local.

Cloud-based testing provides web-based automation platforms that can be accessed from anywhere in the world with good internet connectivity. It is one standard way in which one can achieve native and hybrid types of test automation. And the “automation-from-anywhere” feature is a big advantage. You can run scripts from your test framework on most of the cloud solutions.

Local-based testing involves setting up tools in a test environment and leveraging either emulators/simulators or physical devices to automate testing using popular open-source tools such as Appium, Espresso, or Kif. The additional consideration with on-premise testing is the breadth you can cover, and the time to set up these environments.

Mobile Testing, The Practice

Though the techniques and tools used to automate mobile application testing are complex, we learn a lot from the days of client/server desktop applications. But a few extra mechanisms are required for mobile automation, and these are:

  • Using a mobile test framework such as Espresso or Kif makes it easy for developers and testers to write scripts in the native programming language and opens the door for pair programming;
  • Identify the requirements and categorize them based on the mobile application type—Native, Hybrid or Mobile Web;
  • Identify the scope and device that meets the requirement;
  • Identify an automation tool that will a best fit. Filter the best tools by performing Proof of Concept (POC) that would prove that the automation can produce real results;
  • Design the framework architecture based on the initial requirement.

Test automation strategy runs in parallel with the framework design, which details the technical scope, test environment, and running scripts on emulators/physical device with appropriate automation based approach.

Mobile Device Matrix:

In addition to the testing mechanisms, you have to consider the matrix of device type, device OS, and device browsers to test against. The size of this matrix is significantly larger than in web application testing because of the dependency of device types since each have different screen resolution possibilities and different features. For example, the new thumb print authentication in some iPhones is not available in others, but applications can leverage this feature. Templates for building a mobile automation device matrix require:

  • Operating Systems – customizations, missing libraries, driver issues;
  • Screen Size – rendering issues, usability, missing layouts;
  • Pixel Density – density independence, missing layouts;
  • Aspect Ratio – X,Y calculations, overlapping panels, display issues;
  • System on a Chip (SoC): hardware performance, instruction set, battery signal;
  • Carrier: network protocol, speed, responsiveness, packet loss.

Conclusion

To build a successful test mobile automation strategy, all stakeholders must understand the business value of automation. Further, all teams, including development, need to buy in to the process. Picking the right automation tool and building the right testing environment are typically the most difficult challenges when implementing a successful test automation process. However, you do need to focus on the long-term success of your product, so it’s helpful to address these important questions when starting the process:

  • What are your automation objectives?
  • Will your current development processes need to change?
  • What test environment support do you need?
  • What skills will you need?

Process and Organization + Environment + Technical + Resources + Scope and Roadmap = Test Automation Strategy.

Mobile testing tools are relatively new, and developing an automation strategy specific to your organization ensures that you will get the most business value out of your automation tool. A crucial piece to laying the groundwork for mobile app testing is building a well-balance testing portfolio that includes automated unit testing, integration testing, WebView testing, automated UI testing and exploratory manual app testing of the UI. Appium shows great promise in this area.

Making the leap into mobile testing today is not easy. The good news is a there are many tools that are being developed that support mobile testing efforts, but the biggest part of the challenge is integrating your testing into your existing environment. The important thing to know is that just because it is hard does not mean you should reduce your efforts compared to your web application. This would result in a disconnect of software quality, and could ultimately destroy your mobile application strategy from within.

By Greg Sypolt

Greg Sypolt is a Senior Engineer at Gannett Digital with 10 years of focus on project quality, results, and customer satisfaction while serving in multiple leadership roles. He is an expert in all areas of testing – including functional, regression, API, and acceptance testing – in both the automated and manual realm. Greg is experienced with test strategy planning, including scope, estimates, and scheduling. Greg also assumed the role of Automation Architect while helping convert a large scale, global test team from a manual to an automated focus.

Video: Jason Huggins – Fixing HealthCare.gov, One Test at a Time

April 23rd, 2015 by Amber Kaplan

This week, Sauce Labs co-founder and Selenium creator Jason Huggins came to visit to chat about his leave of absence to help fix HealthCare.gov.  For those who missed his talk at the Selenium meetup, we’re happy to report that we got our hands on a recording.  Check out the video below to watch.

In the video, Jason references a federal “digital playbook” that was written and published following the healthcare.gov overhaul. Point # 10 in the playbook –  “Automate Testing and Deployments” – was suggested and driven by Jason. #TestAllTheThings

 

ABSTRACT

In late 2013, Selenium creator Jason Huggins joined President Obama’s “tech surge” team to help fix HealthCare.gov. In D.C. during the height of the crisis in November and December 2013, Jason had a behind-the-scenes view into a unique period in American history when a website’s quality (or lack thereof) had the attention of the nation, the press, the President, and Congress.

In this talk, Jason will share some of his stories from the HealthCare.gov turnaround and the “HealthCare 2.0″ effort in mid-2014. Jason will talk about the newly created U.S. Digital Services and how it was created out of the original HealthCare.gov crisis. He’ll also cover the U.S. Digital Services Playbook and what the role of automated testing and deployment will be in future U.S. Government projects.

Lastly, Jason will talk about opportunites for how Silicon Valley can help government build effective digital services in the future.

BIO

Jason is a software engineer living in Chicago. He started the Selenium project in 2004 at ThoughtWorks. He later joined Google to work on large-scale web testing for Gmail, Google Maps, and other teams. He left Google to co-found Sauce Labs as CTO to create a cloud-based Selenium service. In late 2013, Jason took leave from Sauce to help with the HealthCare.gov turnaround. He is also the creator of Tapster, a mobile app testing robot that’s been featured in Popular Science, Wired,Tech Crunch, and the MIT Technology Review.

Best Practices in Mobile CI [WEBINAR]

April 22nd, 2015 by Amber Kaplan

Modern organizations today feel immense pressure to deliver better software faster, and this is no different in the mobile space. The best practice of Continuous Integration for web dev has been embraced for years as it is a proven mechanism that accelerates production cycles. However,mobile developers have been slow to adopt CI, despite needing a quick go-to-market plan.

In large part, this is because mobile brings with it a set of unique challenges that make implementation tough. Nevertheless, tools have evolved and mobile development teams now have many options to choose from to implement a solid mobile CI system.

In our next webinar, Kevin Rohling (Emberlight, Ship.io) and Kristian Meier (Sauce Labs) will cover best practices in implementing a mobile CI system and demonstrate how you can easily build, test, and deploy mobile apps.

This webinar will cover: Read the rest of this entry »

Guest Post: All About Mobile Continuous Integration

April 20th, 2015 by Amber Kaplan
mobile_devices_continuous_integration

Image credit: Loris Grillet

CI’s not a new thing. Wikipedia says the phrase was first used back in 1994, way before modern mobile apps. Today it’s commonplace in many dev shops for developers to expect that their code is automatically tested when they commit and even automatically deployed to a staging environment.

For mobile developers though, it’s been a slow road to adopting many of these same practices.  In large part, this is because mobile brings with it a whole set of unique challenges that make implementation tough.  Nevertheless, tools have evolved a lot and mobile dev teams get a lot of value and goodness from having a solid CI system in place. Here are my top 3 reasons for using CI with the mobile products I work on: Read the rest of this entry »

Sauce Labs Continues To Expand Senior Management Team With Two Key Hires

April 16th, 2015 by Amber Kaplan

SAN FRANCISCO, CA – Sauce Labs, Inc., the leading cloud-based web and mobile application testing platform, today announced it has appointed Tom Kress as Vice President of Sales and Lubos Parobek as Vice President of Product. The moves further support Sauce Labs’ momentum, equipping the company to meet the unprecedented demand for its web and mobile app testing platform. Read the rest of this entry »

Guest Post: Functional Testing in 2016 – Forecast

April 15th, 2015 by Amber Kaplan

Massive changes in the development world are good and extreme for devs, but quality assurance (QA) teams are impacted as much, if not more. They may be taking on more tasks, looking at new tools, and thinking about new ways to execute their growing test suites. And looking forward, QA in the future looks much different than it does today. It is moving so fast that the changes – both good and bad – will be even more obvious by next year. Here is what QA looks like in 2016. Read the rest of this entry »

Guest Post: What Automated Testing Really Provides

April 9th, 2015 by Adam Goucher

One of the more fun meme’s on the internet is the ‘correlation does not mean causation‘. One of my favourite examples of this is stop global warming: become a pirate. Application development, and society at large, are full of these [fun] problems.

So what have we incorrectly correlated in the browser automation niche? The number of browser-based scripts and a decrease in test cycle time.

First off, the decision to be ‘done’ testing is actually quite arbitrary. And though the inputs into that decision are often quite uniform, occasionally someone or something will throw a monkey wrench into the mix and people make a guess/take a risk. Therefore, there is no correlation between the number of scripts and test cycle time. Read the rest of this entry »

Speed up Selenium with Application Mocks

April 8th, 2015 by Amber Kaplan

Let’s face it, we all know Selenium is slow. Not the Selenium scripts themselves, but the process of interacting with the operating system and browser drivers, in addition to your application’s own responsiveness. Luckily for us, there’s a great way to speed up functional testing in Selenium by combining a comprehensive parallel testing grid with application mocks. Read the rest of this entry »

Practical Tips + Tricks for Selenium Test Automation [RECAP]

April 7th, 2015 by Amber Kaplan

selenium testing & sauceThanks to all of you who joined us for Practical Tips + Tricks for Selenium Test Automation with expert Dave Haeffner – author of The Selenium Guidebook.

In this webinar, Dave explained the best and most useful tips & tricks from his weekly Selenium tip newsletter (elementalselenium.com).

Topics covered included: Read the rest of this entry »

Applitools Recognizes 16 Influencers + Thought Leaders in Test Automation

April 2nd, 2015 by Amber Kaplan

Test Automation Top Influencers and Thought Leaders_500wSaucesome news in the automated testing world! Our friends at Applitools recognized several Sauce employees and friends in their Top 16 Influencers and Thought Leaders in Test Automation list, including our Co-Founder Jason Huggins, Director of Ecosystems and Integrations Jonathan Lipps, and Selenium expert Dave Haeffner.

Applitools stated, Read the rest of this entry »