Posts Tagged ‘testing’

Automated Mobile Testing with Real Devices

September 8th, 2015 by Ken Drachnik

Mobile testing has long been a manual process – limited by the devices you have close at hand or at best, a painfully slow process in the cloud.  Sauce has long had emulators and simulators to let you speed your tests by automating them in the cloud. Sometimes, however, emulators are not enough and you need to test on real devices.  We are happy to announce the public beta of our Real Device Cloud. That’s right, we’ve put hundreds of Android and iOS devices up in our cloud so you can automate your tests across emulators and real devices with massive parallelism. That means you can now test fast – just break your tests up into manageable bits and run them in parallel. There’s no waiting, there’s no reservation system – they are available and ready to test whenever you are.

We anticipate opening The Real Device Cloud by the end of the month so you can start testing all your mobile apps.  Download the datasheet.

The Real Device Cloud

  • Instant Availability – Get access to the most popular* iOS or Android devices with no waiting
  • Over 80 mobile emulators and simulators
  • Massive Concurrency – Run your tests in parallel to reduce your total test time
  • Integrate with your CI tool of choice – automate all your tests using the top CI tools like Jenkins, Bamboo, Travis, Team City or Circle CI
  • Test Native, Mobile Web and Hybrid apps
  • Access back end databases – that’s right, you can test your app in our cloud and have secure access to your backend data and websites for a true end-to-end test.
  • Full reporting – instant access to videos, screenshots and all the data for your tests so you can analyse your results quickly
  • Enterprise features – account and team management lets you provision test resources as needed and SSO integration means you don’t have to go to IT to add another tester to your account
  • Professional Services and Training – we have people and partners to help you get started with Appium or if you’re already proficient, our experts can help your team become super users

* The Real Device Cloud will open with Apple iPhone 6 and Samsung Galaxy 4S and 5S phones.  We will be adding more device types in the near future.

Sign up for the beta program today.

Can You Test it All? Test Coverage vs. Resources

September 3rd, 2015 by Ashley Hunsberger

During nearly every project I have worked on, the question Can I test everything? always comes up.  The answer is (usually) a resounding NO. Sometimes it’s because of time, sometimes it’s lack of people. How can we still ensure a quality product, even if we can’t cover it all? Sometimes, we have to test smarter.

The usual suspects

The typical scramble to finish testing and get something released is usually (in my experience) a result of one of the following (or a combination thereof):

User stories that are WAY too big.  When user stories are too large, it makes it difficult to break out tasks and identify all the acceptance criteria. They also become more difficult to plan for unforeseen scenarios, and can often blow estimates out of the water.

Complex Workflows. Depending on your feature, the workflow could be very complicated, and it can be difficult to anticipate how a user is actually going to use the product. This makes it more challenging to find every possible scenario for end-to-end tests. Even if your user stories are small, the overall workflow comprising all user stories can still result in missed tests if it is too complex.

Not using Test Driven Development. If you are still living in a world where Development works on their own and throws it over the proverbial fence to QA, you are opening up doors for late surprises to enter, and blocking bugs that hinder your testing progress. (more…)

Should You Have a Dedicated Automation Team Within Your QA Department?

September 1st, 2015 by Israel Felix

If you’ve led or managed QA teams that have included an automation test team, you’ve probably been in a situation where you had to decide whether you should keep them on board. Normally the decision needs to be made when there is a change in leadership, wherein the new management comes with a mandate to consolidate groups and reduce costs. This situation also tends to arise when working with startups or small companies when they are ready to put together or augment their QA teams. So should you have a dedicated automation team?

Typically, there are two camps with regards to dedicated automation teams. There are those who believe that we should have dedicated automation teams, and those who believe that QA engineers should handle manual testing and automation testing. From my experience working in QA within both small and large companies, I almost always prefer to have a dedicated automation team. However, there are a few scenarios where having a QA team that takes on both roles might make sense.

Time to Market

For automation to be done right, it needs to be a full-time job. From developing the framework and creating the libraries and scripts for different platforms to executing and debugging failures — it will all simply consume too much of an engineer’s time and compromise the actual testing and release date. As you already know, time to market and keeping a release on schedule is top priority, so testing needs to get done, no matter what. (more…)

New! Support for OS X 10.11 and iOS 8.3, 8.4 & 9.0

August 12th, 2015 by Ken Drachnik

El Capitan 200 px wideIn our continuing effort to make Sauce Labs the best place to test, we’ve just added some additional OSes and Browsers. In addition to Windows 10 and the Edge Browser we announced last week, today we are extending the platforms we support to include released and upcoming iOS and OS X operating systems:

OSX 10.11 El Capitan (beta)
iOS 8.3 and 8.4
iOS 9 (beta)
Safari 8.0.7

To use any of these platforms visit our platforms configurator.

When we ask our users what it is they love about Sauce one of the most common responses is the wide variety of platforms we provide our users. In an ecosystem of increasing complexity, we know our customers have a lot of devices and platforms they need to test against. Our humble aspiration is to make your life a just little bit easier by having the platforms you need when you need them.

Sign in to start testing

Appium + Sauce Labs Bootcamp: Chapter 2, Touch Actions

June 15th, 2015 by Isaac Murchie

One aspect of mobile devices that needs to be automated in order to fully test applications, whether native, hybrid, or web, is utilizing gestures to interact with elements. In Appium this is done through the Touch Action and Multi Touch APIs. These two APIs come from an early draft of the WebDriver W3C Specification, and are an attempt to atomize the individual actions that make up complex actions. That is to say, it provides the building blocks for any particular gesture that might be of interest.

The specification has changed recently and the current implementation will be deprecated in favor of an implementation of the latest specification. That said, the following API will remain for some time within Appium, even as the new API is rapidly adopted in the server.

Touch Actions

The Touch Action API provides the basis of all gestures that can be automated in Appium. At its core is the ability to chain together _ad hoc_ individual actions, which will then be applied to an element in the application on the device. The basic actions that can be used are:

  • press
  • longPress
  • tap
  • moveTo
  • wait
  • release
  • cancel
  • perform

Of these, the last deserves special mention. The action perform actually sends the chain of actions to the server. Before calling perform, the client is simply recording the actions in a local data structure, but nothing is done to the application under test. Once perform is called, the actions are wrapped up in JSON and sent to the server where they are actually performed! (more…)

Appium + Sauce Labs Bootcamp: Chapter 1, Language Bindings

June 1st, 2015 by Isaac Murchie

Appium logo w- tagline {final}-01Welcome to the first in our new series, Appium + Sauce Labs Bootcamp. This first chapter will cover an overview of Appium and its commands, demonstrated with detailed examples of the Java and Python language bindings.  Later we will follow up with examples in Ruby. This series goes from fundamental concepts to advanced techniques using Appium and Sauce Labs. The difficulty is Beginner->Advanced. (more…)

Video: “Eliminate Rollbacks” Talk by Neil Manvar

May 6th, 2015 by Bill McGee

Missed Velocity Conf SF? We’ve got you covered. Check out this talk by Neil Manvar, Sauce Lab’s new Professional Services Lead. View “Eliminate Rollbacks” below and let us know what you think in the comments.

New Web and Mobile Testing Trends Research Report

February 11th, 2015 by Bill McGee

Sauce Labs recently partnered with Dimensional Research to conduct an independent survey of technology professionals to understand current trends in web and mobile testing. With the proliferation of devices, browsers and operating systems, and the rapid adoption of modern software development methodologies, we wanted to see how development teams are balancing these complex user environments with current testing resources and the impact agile methodologies have had on software testing. Perhaps not surprisingly, 94% of respondents who have adopted continuous integration (CI)  and other agile methodologies said they’ve changed their approach to testing, and for the better – they’re testing earlier and more often, and are able to do more of all types of testing. You can see the full report here – happy reading!




How Leverages Appium for Mobile Test Automation

July 1st, 2014 by Bill McGee

We love this blog post written by Quentin Thomas at HotelTonight! In it, he explains how they use Appium to automate their mobile tests. He also walks readers through specifics, such as the RSpec config helper. Read a snippet below.

Thanks to the engineers at Sauce Labs, it is now possible to tackle the mobile automation world with precision and consistency.

Appium, one of the newest automation frameworks introduced to the open source community, has become a valuable test tool for us at HotelTonight. The reason we chose this tool boils down to Appium’s philosophy.

“Appium is built on the idea that testing native apps shouldn’t require including an SDK or recompiling your app. And that you should be able to use your preferred test practices, frameworks, and tools”.

-Quentin Thomas, HotelTonight, June 17, 2014

To read the full post with code, click here. You can follow Quentin on Twitter at @TheQuengineer.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Guest Post: Automating Mobile Testing at Gilt – A Follow Up

June 19th, 2014 by Bill McGee

This post comes from our friends at Gilt, who are using Appium to automate their mobile testing.  Check out their first post here.

It’s been a while since we’ve posted in detail about our automation and testing process here on the Mobile Team at Gilt. Much has changed in the past few months, and the past couple of weeks in particular have been pretty exciting on the testing front. First off, we had the opportunity to give a talk about how we approach native mobile testing, to a packed house at the Brooklyn iOS Meetup. Second, Appium 1.0 – now already 1.1 – was officially released! Cue the champagne and fireworks! In seriousness though, the Appium milestone release is a pretty big deal for us.

Some History

We’ve written briefly about Appium before, and how it fits in with our overall process. About a year ago, Appium popped up on our radar here at Gilt after one of our engineers saw a talk about it at SeConf. We were intrigued and did a bit of experimenting, and after discovering that we’d be able to leverage some of our existing Selenium infrastructure, as well as our in house Selenium talent, we deciding to adopt Appium as our testing framework of choice.

Appium had been in development for some time by this point, but it was still early days for the project, and it had some rough edges. Tag names were inconsistent, and the XPath locator strategy was pretty sad. We’ll get into that later…

Appium’s pace of development was really fast, and things changed regularly. Often the changes would be drastic and would break our tests.
Stability also seemed to fluctuate. On some versions, things were terrific, and our tests would run smoothly. Other times, tests would fail or crash for no apparent reason. The failures weren’t always Appium’s fault, but they were still pretty frustrating. Once or twice we wondered why we were even bothering and considered switching back to KIF. We persisted though, and we’re really happy we did, because Appium has grown up!

So,  What’s All The Fuss About?

Appium now implements the defined part of the Mobile JSON Wire Protocol. This is basically an extension to the selenium wire protocol, that provides support for mobile specific actions. Desired capability keys seem to have been finalized by this spec, which is great, since these have changed a couple times over the past year, causing a couple of headaches. It’s also good to see some bigger companies getting involved here.

For us though, the toughest issue has been stability. Countless hairs have been turned grey due to tests failing or crashing for no apparent reason. While we’ve jumped through a variety of other hoops in our quest to solve the overall flakiness issue (page objects, judicious waiting), Appium has made huge strides in this area. This is no small achievement, given that Appium is really proxying commands to UIAutomation, which itself is pretty fragile and prone to random crashing.

We’re also super happy to see that force-quit-instruments no longer shuts test sessions down during long waits or element searches.
Most surprisingly, page source is now XML based. XML? It’s like it’s 1999 all over again! Actually this change has some pretty amazing benefits. The biggest of which is that we can now perform “real” XPath queries on our page source, and Appium is now using a compliant XPath parser. Hooray! We complained before about Appium’s XPath engine being pretty lousy. In olden times, the XPath engine was regex based and would search for the desired element(s) in a JSON representation of the page source. Dreadful stuff… Needless to say, this was inefficient and only worked correctly with “just right” XPath queries. This basically left Appium users with only two legitimate locator strategies. We’re super happy to see that this has been resolved.

Too Good To Be True?

We’re generally really happy with the way Appium works, but there are a handful of issues that still cause some pain for us. Error reporting is not Appium’s strong suit. While overall reliability is orders of magnitude better in the latest release, it’s still incredibly hard to identify the exact problem when failures occur. Integrating Appium with your Xcode workflow is left up to you, and while this isn’t terribly hard, it would be nice to see tighter Xcode integration. I still stand by our previous advice of steering clear of any non-standard UI tricks and hacks. Minimize the use of overlays, and do whatever you can to avoid passthrough views. If anything, Appium will push you to strive for UI simplicity and to get serious about accessibility. The proposed touch gestures API might provide some flexibility here, but we’ll have to wait and see.

Coming Soon

We’re still busy getting to grips with the changes brought by this milestone release. The increased interest in the the Mobile JSON Wire Protocol by some bigger companies has pushed our confidence in Appium up a couple notches. While the spec is not yet complete, it does propose some really exciting features down the line. Notably, we’re really excited by the prospect of server-side waits. The way we manage waiting for elements has had a huge impact on the reliability and speed of our tests, and so it’s really exciting to see this part of the process getting some love.

The Bottom Line

Almost all reliability issues that plagued us in earlier versions appear to have been resolved. We’re now able to spend more time working on tests, and much less time wrestling with Appium and Instruments. Prior to this release, our tests were totally untrustworthy. The changes made leading up to the 1.0 release, combined with better test practices on our part, have dramatically boosted our confidence in our automated tests. Test results are now taken seriously and are becoming an invaluable aid when performing QA work on our apps. We’re really excited to see how the Mobile JSON Wire Protocol evolves, and what comes next for Appium.

By Matt Isaccs of Gilt Groupe for Sauce Labs

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.