Sauce Labs Named a Contender Among Mobile Test Automation Tools

April 28th, 2016 by Bill McGee

We are happy to announce a new industry citation for Sauce Labs: our inclusion in “The Forrester Wave™: Mobile Front-End Test Automation Tools, Q2, 2016.” The Wave is an influential report by leading research firm Forrester Research, and Sauce Labs was one of 10 companies selected for inclusion.

This Wave is based on an evaluation of 40 criteria across 10 vendors that considers how each provider measures up to help application development and delivery. Sauce Labs was among vendors cited as a “Contender” in the report. On describing Sauce, the report states:

Sauce Labs is the open source champion, offering comprehensive target app types … Sauce Labs successfully embraces the bring-your-own-tools (BYOT) and pick-your-favorite-language approaches to mobile app testing, which should make most developers happy. Focusing on automation and CI, it offers a robust, cloud-only testing solution that includes virtual machines for testing web and mobile applications as well as real devices. It satisfies security-conscious environments through a secure connection back into the data center using Sauce Connect.

More than anything, we are beyond excited that our Real Device Cloud (RDC) was included in such a significant report. Especially as we know Forrester went through an exhaustive evaluation of Sauce Labs that looked at factors including product fit, customer success, and Forrester client demand.

We believe this report provides Sauce Labs with huge visibility to potential new users, as many North America Fortune 1000 companies are Forrester clients. We are excited to be a part of the growing Mobile testing market, we are encouraged by the recognition of our strategy and roadmap, and we are thrilled with the momentum the Sauce Labs Real Device Cloud has gained in a short amount of time.

Want to learn more about automated mobile testing? Download our free report, “Automated Mobile Testing Requires Both Real Devices and Emulators“. 

Appium 1.5.2 Released on Sauce

April 26th, 2016 by Isaac Murchie

We are pleased to announce the release of Appium 1.5.2 through npm and on Sauce Labs! This is a bug fix release that deals with many issues with 1.5.1, including a major bug where large Android applications would cause Appium to run out of memory and die.

General

  • deprecated --command-timeout. Use newCommandTimeout desired capability instead
  • ensure implicit wait can be set through timeout method
  • add better logging for EPIPE errors

iOS

  • make sure ipa files are handled correctly for installing on real devices
  • ensure that existing SafariLauncher on device is used instead of rebuilding and reinstalling
  • fix issues with getting webview contexts on real devices
  • add full timeout support through timeout method
  • make sure Xpath searches respect implicit wait timeout
  • make sure bare Instruments process arguments are accepted

Android

  • fix failure when apk file is too large
  • re-implement setting geolocation so it does not use Telnet
  • add support for Chromium browser
  • fix issues with flick
  • fix bug where touch action release would throw an error
  • fix bug in later Android SDK version where noticing a newly started avd would fail
  • implement autoWebviewTimeout

Mobile vs. Web: Which is Harder to Test?

April 19th, 2016 by Joe Nolan

Have you ever worked on a web-based test team and switched to a mobile team and wondered if your life is about to get easier or harder? There are significant differences between testing mobile vs. web, and yes, one is MUCH harder than the other. Want to guess which one? Read on and see if you guessed correctly.

Let’s Compare

The table below shows the different facets of testing and where its execution is most challenging. (more…)

test.allTheThings() – Announcing General Availability of Real Device Cloud

March 31st, 2016 by Ken Drachnik

After months of beta testing with customers, we are announcing general availability for our Real Device Cloud today.  With the ability to test websites, native apps, and hybrid apps on both iOS and Android devices, we enable enterprise customers to easily scale their CI/CD testing needs on the most popular devices. By providing large numbers of each device type, our customers will not have to wait in a queue for their tests to run on a real device, thus speeding up the pace of testing, especially when running concurrent parallelized tests.  In conjunction with our mobile emulators and simulators, we now provide the most comprehensive automated mobile testing platform in the market, covering both web and mobile app testing needs across a variety of device types. After running your tests, view the results on our test details pages, with videos of each test, screen shots, logs and commands so you can resolve issues quickly and iterate on your app more frequently.  

To request a quote for real devices, contact your account executive.

Mobile Testing on Sauce Labs Includes:

  • Instant Availability – Get access to the most popular iOS or Android devices with no waiting, no queues and no reservations system.
  • Android emulators and iOS simulators covering over 140 device-OS combinations.
  • Massive Concurrency – Run your tests in parallel to dramatically reduce your total test time.
  • Integrate with your CI tool of choice – automate all your tests using the top CI tools like Jenkins, Bamboo, Microsoft VSTS, Travis, Team City or Circle CI.
  • Test native apps, hybrid apps, and mobile web –  all on the same platform.
  • Security – test with Sauce Connect, allowing your codebase to stay behind your firewall while utilizing Sauce’s extensive collection of OS platforms, browsers, and devices.
  • Pinpoint issues quickly with full reporting – instant access to videos, screenshots and all the data for your tests so you can analyse your results quickly.
  • Enterprise management features – account and team management lets you provision test resources as needed and SSO integration means you don’t have to go to IT to add another tester to your account.
  • Professional Services and Training – we have professional consultants and partners to help you get started with Appium and Selenium or if you’re already proficient, our experts can help your team become super users.

 

For more information, visit our Automated Mobile Testing Page

Appium 1.5.1 Released on Sauce

March 29th, 2016 by Isaac Murchie

We are happy to support the newly-released Appium 1.5.1. This release fixes a
number of issues with 1.5.0, including one bug that prevented some frameworks
from correctly polling for status during Safari tests.

General

  • allow `platformName` to be any case
  • Windows process handling is cleaned up
  • Desired capabilities `language` and `locale` added

iOS

  • iOS 9.3 (Xcode 7.3) support
  • Fix handling of return values from `executeScript` in Safari
  • Don’t stop if Instruments doesn’t shut down in a timely manner
  • Escape single quotes in all methods that set the value on an element
  • Allow custom device names
  • Make full use of process arguments to Instruments
  • Pass `launchTimeout` to Instruments when checking devices

Android

  • Make use of `–bootstrap-port` server argument
  • Fix `keystorePassword` capability to allow a string
  • Fix handling of localization in Android 6
  • Use Appium’s unlock logic for Chrome sessions
  • Make sure reset works
  • Make unlock more reliable for later versions of Android
  • Allow Xpath searching from the context of another element
  • Make full use of process arguments to adb
  • Better error messages when ChromeDriver fails to start

Reverse Engineering App Strategies

February 25th, 2016 by Joe Nolan

When interviewing for a new job on a mobile development team, information about their development strategy, future feature plans, technology and quality focus is hiding in plain sight — if you know where to look.

You’ve done your due diligence by reviewing sites like Glassdoor, Monster, LinkedIn, and Fortune, so you have a general idea of the company culture and how the world perceives it. But, as a technologist, it is in your best interest to dig deeper and interpret the clues that are readily available about the development culture you wish to join.

A friend of mine was recently looking for a position on a mobile team and turned to me for advice, knowing that I manage a mobile QA team. We put on our detective hats and went to work. (more…)

Measuring Mobile App Quality

February 18th, 2016 by Joe Nolan

Sure you see your bug reports in JIRA, but how do you actually know the level of quality in your apps and processes? Bug count metrics are a great starting point, but if you really want to know if your team is producing a quality app and improving their internal processes, you need to look to other tools to see how your product is trending.

Know Thyself

If your internal processes and test coverage are poor, most likely this will translate into a poor app. Get your house in order with the following:

  • Start by watching your automated build and test results, using a tool like Sauce Labs
  • Use a tool like Datadog to understand what you want to measure, and monitor trends in build failures based on automated test results, and analyze how long it takes to deploy. You can use this information to determine direct and indirect impacts to the different teams.
  • Determine the important things you want to include in your CI and Automated Test coverage. Have a percentage goal of coverage to strive for.
  • Monitor your bug backlog. Is it accumulating per release?
  • Keep track of your system availability for customers. Use a tool like Crashlytics to measure who is using the application, real time data about the app’s performance and problems.
  • Set performance tests baselines, using tools such as JMeter to test your APIs, and Charles to test your page load times.
  • How are the scrum teams doing? Are they improving their ability to produce more story points per sprint as they get used to working together and with their features?

(more…)

The Importance of Mobile Testing

November 12th, 2015 by Greg Sypolt

It is a digital world, and all Generations have embraced the mobile revolution. It’s no surprise that millennials are leading the way. I saw a statistic recently that 85% of Millennials owns a smartphone. Next time you are waiting in line, take a look around.  Statistics don’t lie.

Source: http://www.statista.com/chart/3760/mobile-etiquette/

Source: http://www.statista.com/chart/3760/mobile-etiquette/

The mobile revolution has only just begun. The importance of having a mobile testing strategy is critical to our app releases.

What makes a great mobile app?

There are multiple elements that can influence a great (or bad) mobile app. The team works so hard developing an app (from idea, planning, writing code, and testing), the last thing you want to hear is negative feedback instantly after releasing the app. As long as the focus is on simplicity, understanding your users (audience), and the performance of your app, you are heading in the right direction. (more…)

Assessing Risk for a QA Mobile Project

October 22nd, 2015 by Joe Nolan

My niece and her kids were visiting from out of town last week. Somehow I ended up in charge of the two boys, six and eight years old. It all started nice and quiet, everyone playing on their devices. Next thing I know they are wrestling. Heads are just missing table corners and I’m freaking out! Don’t they know the risks?

As a lead on a QA team developing mobile apps, I am constantly evaluating risk. What is the risk to the schedule? To the end users? While nobody (hopefully) will be bloodied like two boys playing, risks need to be accounted for.

Identify the Risks

Risks are evaluated differently by different people, and one person cannot identify all of the risks. So, how does risk identification happen? (more…)

iOS 9.1 Beta Support and Speed Improvements

October 16th, 2015 by Ken Drachnik

With a number of reported problems on iOS 9.01 and 9.02, Apple is in beta with a 9.1 version that is supposed to fix issues and become the stable release.  We have iOS 9.1 beta ready now, so you can start testing and be ready for the general release predicted for late October or November.  Check out all our browser / OS combinations at our platform configurator.

At Sauce we are constantly improving our service by making it easier to integrate with CI systems, upgrading our UI and, lately, improving reliability and speed.   We’ve just finished part one of a project to speed up the generation of VMs so you can test even faster.  Our Windows VMs are now faster than ever, so check ’em out.