Posts Tagged ‘testing’

How HotelTonight.com Leverages Appium for Mobile Test Automation

July 1st, 2014 by Amber Kaplan

We love this blog post written by Quentin Thomas at HotelTonight! In it, he explains how they use Appium to automate their mobile tests. He also walks readers through specifics, such as the RSpec config helper. Read a snippet below.

Thanks to the engineers at Sauce Labs, it is now possible to tackle the mobile automation world with precision and consistency.

Appium, one of the newest automation frameworks introduced to the open source community, has become a valuable test tool for us at HotelTonight. The reason we chose this tool boils down to Appium’s philosophy.

“Appium is built on the idea that testing native apps shouldn’t require including an SDK or recompiling your app. And that you should be able to use your preferred test practices, frameworks, and tools”.

-Quentin Thomas, HotelTonight, June 17, 2014

To read the full post with code, click here. You can follow Quentin on Twitter at @TheQuengineer.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Guest Post: Automating Mobile Testing at Gilt – A Follow Up

June 19th, 2014 by Amber Kaplan

This post comes from our friends at Gilt, who are using Appium to automate their mobile testing.  Check out their first post here.

It’s been a while since we’ve posted in detail about our automation and testing process here on the Mobile Team at Gilt. Much has changed in the past few months, and the past couple of weeks in particular have been pretty exciting on the testing front. First off, we had the opportunity to give a talk about how we approach native mobile testing, to a packed house at the Brooklyn iOS Meetup. Second, Appium 1.0 – now already 1.1 – was officially released! Cue the champagne and fireworks! In seriousness though, the Appium milestone release is a pretty big deal for us.

Some History

We’ve written briefly about Appium before, and how it fits in with our overall process. About a year ago, Appium popped up on our radar here at Gilt after one of our engineers saw a talk about it at SeConf. We were intrigued and did a bit of experimenting, and after discovering that we’d be able to leverage some of our existing Selenium infrastructure, as well as our in house Selenium talent, we deciding to adopt Appium as our testing framework of choice.

Appium had been in development for some time by this point, but it was still early days for the project, and it had some rough edges. Tag names were inconsistent, and the XPath locator strategy was pretty sad. We’ll get into that later…

Appium’s pace of development was really fast, and things changed regularly. Often the changes would be drastic and would break our tests.
Stability also seemed to fluctuate. On some versions, things were terrific, and our tests would run smoothly. Other times, tests would fail or crash for no apparent reason. The failures weren’t always Appium’s fault, but they were still pretty frustrating. Once or twice we wondered why we were even bothering and considered switching back to KIF. We persisted though, and we’re really happy we did, because Appium has grown up!

So,  What’s All The Fuss About?

Appium now implements the defined part of the Mobile JSON Wire Protocol. This is basically an extension to the selenium wire protocol, that provides support for mobile specific actions. Desired capability keys seem to have been finalized by this spec, which is great, since these have changed a couple times over the past year, causing a couple of headaches. It’s also good to see some bigger companies getting involved here.

For us though, the toughest issue has been stability. Countless hairs have been turned grey due to tests failing or crashing for no apparent reason. While we’ve jumped through a variety of other hoops in our quest to solve the overall flakiness issue (page objects, judicious waiting), Appium has made huge strides in this area. This is no small achievement, given that Appium is really proxying commands to UIAutomation, which itself is pretty fragile and prone to random crashing.

We’re also super happy to see that force-quit-instruments no longer shuts test sessions down during long waits or element searches.
Most surprisingly, page source is now XML based. XML? It’s like it’s 1999 all over again! Actually this change has some pretty amazing benefits. The biggest of which is that we can now perform “real” XPath queries on our page source, and Appium is now using a compliant XPath parser. Hooray! We complained before about Appium’s XPath engine being pretty lousy. In olden times, the XPath engine was regex based and would search for the desired element(s) in a JSON representation of the page source. Dreadful stuff… Needless to say, this was inefficient and only worked correctly with “just right” XPath queries. This basically left Appium users with only two legitimate locator strategies. We’re super happy to see that this has been resolved.

Too Good To Be True?

We’re generally really happy with the way Appium works, but there are a handful of issues that still cause some pain for us. Error reporting is not Appium’s strong suit. While overall reliability is orders of magnitude better in the latest release, it’s still incredibly hard to identify the exact problem when failures occur. Integrating Appium with your Xcode workflow is left up to you, and while this isn’t terribly hard, it would be nice to see tighter Xcode integration. I still stand by our previous advice of steering clear of any non-standard UI tricks and hacks. Minimize the use of overlays, and do whatever you can to avoid passthrough views. If anything, Appium will push you to strive for UI simplicity and to get serious about accessibility. The proposed touch gestures API might provide some flexibility here, but we’ll have to wait and see.

Coming Soon

We’re still busy getting to grips with the changes brought by this milestone release. The increased interest in the the Mobile JSON Wire Protocol by some bigger companies has pushed our confidence in Appium up a couple notches. While the spec is not yet complete, it does propose some really exciting features down the line. Notably, we’re really excited by the prospect of server-side waits. The way we manage waiting for elements has had a huge impact on the reliability and speed of our tests, and so it’s really exciting to see this part of the process getting some love.

The Bottom Line

Almost all reliability issues that plagued us in earlier versions appear to have been resolved. We’re now able to spend more time working on tests, and much less time wrestling with Appium and Instruments. Prior to this release, our tests were totally untrustworthy. The changes made leading up to the 1.0 release, combined with better test practices on our part, have dramatically boosted our confidence in our automated tests. Test results are now taken seriously and are becoming an invaluable aid when performing QA work on our apps. We’re really excited to see how the Mobile JSON Wire Protocol evolves, and what comes next for Appium.

By Matt Isaccs of Gilt Groupe for Sauce Labs

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

 

[Live Panel] Mobile Testing with Appium: Recap

June 17th, 2014 by Amber Kaplan

‘Appy Tuesday! As you may remember, battled-tested Appium 1.0 [Orion] was released on May 2 right here at Sauce Labs HQ. Following the release, we held a live panel-style webinar with core Appium committers Jonathan Lipps, Matthew Edwards, and Dan Cuellar so that they could answer questions about Appium’s philosophy,  the roadmap ahead, best practices for automating mobile tests, and deliver an overview of the release.

To listen to the webinar recording, click here. You can also check out the webinar slides below.

For a quick tour of what’s new with Orion (version 1.o), click here. We’ve also included some follow up questions with answers below, from the Appium team.

Happy Testing!

appium_logo_final

Follow-up Q&A


Why Appium? What distinguishes it from the competition?

The main advantage of Appium is that it’s cross-platform, meaning it will work on iOS, Android, Firefox, OS, etc. You can test in any language; and you don’t have to modify your app. Plus, you can use the selenium protocol that you’re already familiar with from web testing. And on the nice-to-have spectrum, Appium has a great community around it; plus it’s free and open source.

Here’s a quick look at the competition currently:

  • Instruments – Disadvantage? One language. All done in advance. Can’t talk to outside frameworks.
  • uiautomator – This works well; it also has instruments. But the language must be Java. tightly integration with app. Appium has more advanced features such as changing emulator language or xpath locators, however.
  • iOS driver – main disadvantage, small community. not widely used.
  • Selendroid – great project that appium supports. based on instruments, works on older android devices. more limited capabilities compared to uiautomator.
  • KIF – modifying your app, writing objective c into your app. rule #1 is don’t mess with the code. takes it one degree of separation from what the customers are going to see. there are some advantages to doing that. more tight integration, able to do some types of testing that appium can’t
  • Monkey Talk – low level. not the richness of appium

Does Appium work with mobile web and native apps?

Appium’s vision is to help automate everything mobile – including mobile web apps and native apps. It will continue to support Safari on iOS and Chrome on Android to use Appium to test mobile web and native apps. We are open to supporting other browsers.

Does it work with Sauce Labs? How to get started with Appium? This is the first seminar I am attending about this

Appium works on Sauce Labs. To get started checkout the documentationtutorials, and sample code.

How should you handle flaky network connection simulation?

There definitely are tools. Proxies to change what your computer is doing.

Linux/OS X – ipfw – firewall rule tool
Windows – fiddler – javascript for writing rules

Appium also talks over TCP so don’t apply rules to 4444 or 4723.

Automating pre installed apps?

Android works well. UiAutomator allows you to test all apps on the device, on emulators and real devices. You can even go back and forth between apps.

On iOS, you can test in built in apps such as settings only on the simulator. On real devices, apple limits you to testing only your app.

Emulators vs real devices: which is better for testing?

Watch the GTAC presentation here
Check out how Google does their mobile testing here

Then ask yourself, what are you writing your automation for?

  • If it’s for business logic, simulators and emulators are probably fine.
  • If it’s for performance and crashes, consider real devices.

Overall, however, simulators and emulators are a highly effective way to save a lot of money.

I’m joining a new company. How can I convince them to buy into Appium? 

You can use any language and test framework. It’s very easy to take existing selenium test infrastructure and use it for a proof of concept appium test cases. Fastest way to get started with mobile automation. Leverage existing languages and test frameworks that you use. Write a few small test cases then show it off as a demo. When people see appium running for the first time, it’s a pretty amazing experience.

What are the desired capabilities of Appium (versus Selenium)?

The capabilities were updated to match the mobile version of JSWP. Selenium wasn’t designed for mobile, so we worked with the Selenium project to define a new set of capabilities that support mobile.

  • platformName – ios/android
  • platformVersion – 7.1/4.4
  • deviceName – what kind of device we’re talking about iPhone Simulator/Nexus S

Documented well in the migration guide for appium 1.0.

If the same app is written for both platforms, how can I ensure a 100% cross-platform test?

The idea of writing one test for both platforms is the holy grail; but it requires the app be extremely similar. Even with xamarin and the same language for both, it depends on how similar the app is on both platforms.

It’s not about having 100%. Instead,  have a small core that’s different, and then reuse the entire ecosystem about the test code. Everything can be the same except for a single definition file that encapsulates the differences.

Does Appium have significant advantages over webdriver for testing web apps on mobile devices?

Yes! Webdriver doesn’t provide drivers for mobile web apps or native apps. A tool like Appium is the only way you can automate mobile web apps currently.

Interestingly, the Selenium project recommends that you use a project like Appium in order to do this kind of automation. When you’re automating web apps using Appium, it’s just as though you’re using Selenium since there’s no difference in the code you write.  Appium is simply providing the backend instead of the Selenium jar.

What is a short explanation as to what happens when an external application/activity is launched on Android/iOS?

It’s different on each platform. On Android, it’ll be as if the user launched it.

iOS has sandboxed enviornment, if your app opens a 3rd party app then the automation session will be shut down on you. That’s something that only Apple will be able to fix.

What is the release schedule for Appium?

The release process is becoming more structured. The github milestone tracker is being used with estimated dates. They will change.

The project planning is based around this list of issues assigned to a milestone. Feedback is used to prioritize.

How do new versions of Android/iOS impact Appium?

We’re trying to get better at using the beta. For example, we were using iOS 7.1 beta months before it was officially released. We’re able to anticipate issues before release and quickly add official support.

If you try Appium on a new release of Android/iOS, make sure to report issues.

What are Mavericks issues with Appium?

The latest OS X and Xcode release works well with Appium. If there are issues, make sure to open issues on GitHub.

How do I execute parallelization with Appium?

Use Sauce Labs to run in parallel on iOS. Sauce handles the virtualization of OS X for you. You can either maintain a bunch of OS X machines or let Sauce handle it for you.

With Android, it’s easier to run multiple emulators on your own system. If you’re looking to scale, then Sauce Labs is a big win.

Can you have multiple devices connected to one Appium server?

You can run one Appium server per device, on different ports. On iOS, it’s only one device per system due to Apple limitations.

Does Appium support Ruby, Python, C#, … etc?

Yes.

Appium is an open source source tool, but is there a support team that we can contact directly contact in case of issues?

Appium support is similar to how Selenium is supported. There’s a discussion group for questions and an issue tracker on GitHub for reporting bugs. If you have a commercial contract with a company that supports appium, such as Sauce Labs, then they have their own support channels.

Does Appium have the same approach as WebDriver/page objects?

Yes exactly. If you know how to write webdriver code and use page objects, then they’re applied in the same way. All of the concepts transfer.

Are there any hard button clicks supported in latest Appium versions for iOS devices ? I hope Android devices support this.

Appium makes the full UI Automation JavaScript API available as provided by Apple. You can click on buttons within the app. For special buttons such as home, there are work arounds. If you have a specific button in mind, I recommend asking on the appium discussion list.

Android has full support for clicking all buttons. The underlying methods are listed in the UiDevice documentation. To click the back button, for example, the standard selenium command is used. For arbitrary keys, there’s keyevent which is being renamed to press_keycode .

Are there plans to add OCR or image recognition support to appium?

There’s an open issue for FindByImage. Appium supports taking screenshots so you’re free to perform any image recognition based on that data.

Are there any tutorials available as to how to use an existing test automation system (in my case rspec/capybara) with appium?

Appium has code samples on GitHub that include an rspec example. There’s also a tutorial which covers Ruby and Java on appium.io. Documentation is available.

Are you considering hosting a conference about Appium in the near future? say 3 months down the road. Would be nice to have hands-on work shops etc.

If you’re interested in an Appium conference, read this post on the discussion list.

Are you planning to have scheduled releases, like every quarter or every six months, so we know when to expect Appium releases?

Upcoming appium releases along with estimated dates are tracked as milestones on GitHub. For the latest bug fixes, running from source is an option.

Can appium test cases run in parallel on different devices?

Running tests in parallel works fine for Android. On iOS, Apple has limited automation to work on one device at a time. The best way to run at scale with Android Emulators & iOS Simulators is to setup a selenium grid using virtual machines. Sauce Labs provides the easiest way to run tests in parallel on Android and iOS.

Is automation possible without having the Xcode of  an app?

On iOS, you’re expected to have Xcode installed along with a simulator build of the app. A non-simulator build will not work on the iOS simulator. If Xcode isn’t installed then the automation will not work on iOS.

Can I use Appium to test web apps on mobile devices?

Yes. Appium fully supports testing web apps on mobile devices using Chrome on Android and Safari on iOS.

Can we use web application scripts used for Automation in Appium for native app as well?

Reusing test code is going to be a challenge when comparing web automation to native app automation. If you follow the page object pattern, then it’s entirely possible to share some code between the web and native app. Common technology such as test language, results reporting, and parallelization strategy can all be the same.

Can we do gestures on webview for Appium and not use webdriver since gestures are broken for iOS and Android?

The APIs for advanced gestures are currently limited to native apps. Standard WebDriver commands should work inside webviews for gestures.

Can you please provide a short explanation as to what happens when an external application/activity is launched during an Appium session (e.g., the contacts activity in android, or maps in iOS).

The idea behind UI automation is to test the app as a user would. The simple answer to this question is that it’s the same as if a user installed the app and then launched it. For more details, checkout the debug log of the Appium server as it contains exactly what’s going on.

Do I need to be an expert in Selenium WebDriver to effectively use Appium? In other words, is Selenium WebDriver experience a prerequisite for using Appium?

You need to learn WebDriver to use Appium effectively because that’s how Appium works. If you’re already familiar with uiautomator and UI Automation, then it’s possible to use Appium without understanding the details of WebDriver.

If you look at the sample code, the amount of WebDriver knowledge required is not at the expert level.

Do you have known issues when there are multiple devices connected on the same Appium server? We use 6 devices connect to the same Appium server.

To test with more than one Android device locally, you need to have one Appium server per device. For iOS, Apple limits automation to one device at a time. Sauce Labs enables parallel iOS automation by using OS X in a virtual machine.

Does Appium allow one to automate render.js webapps on mobile web for iOS and Android?

Yes, Appium supports automating web apps on iOS and Android.

Does Appium have significant advantages over WebDriver for testing straight up web apps on mobile devices?

Appium is one of the only ways available today to use WebDriver to test mobile apps on mobile devices. WebDriver alone doesn’t support mobile.

for Android, Appium doesn’t detect element id all the time,  so I’ve had to use uiautomatorviewer. But uiautomatorviewer fails while using Appium. Any tips?

Make sure to end any existing Android automation sessions before using uiautomatorviewer. Once you have identified the elements using the viewer tool, then you can update the tests and run them via Appium.

Another workflow is to use the ruby_console to dynamically identify the elements and then interact with them before updating the test code.

Given that I have an app which makes online data exchanges through APIs, and as I want to write reproducible functional tests with Appium, how can I set up an environment to mock my app’s webservices calls in my testing scenarios?

Mocking an app’s webservice calls is a general testing issue and is unrelated to appium. I suggest researching appropriate mocking solutions for your language to see how others have accomplished this.

I have a good working knowledge of Selenium WebDriver 2.0 and of using Page Objects. Does Appium also has the same approach since Appium is using JSON driver behind the scenes?

Yes, the same approach is used.

Hi, I understand that Appium creates only one session for iOS as Apple UIAutomator does not support multiple iOS device support in parallel, whereas in Android, multiple devices are supported and hence each session maps with one devices. Is that correct?

Yes, running in parallel is currently limited to Android due to Apple’s limitations. A work around is to use Sauce Labs as they overcome the one device per machine via virtualization.

How is the support for scrolling going to improve? Currently it is very flaky to scroll up or down.

Properly writing scrolling code is a challenge. I’ve found using explicit waits helps resolve flakiness. On iOS it’s a bit easier because automation of invisible elements is possible. For Android, we’re limited to visible elements. I use complex_find on Android to scroll to elements. A better way is coming soon in Appium that’s easier to use.

Please send some examples of  the multi action gesture API role in Appium?

Check out the unit tests for the various Appium client bindings to see examples of the gesture api.

How do I handle situations such as when Appium just fails, saying Javascript fail error (sendkeys throwing js error on line 68, failed, or my R&D says that Appium was not able to get the page source and that it may be because of a lot of data)?

Please report the issue on GitHub along with a way to reproduce the failure. On iOS I’m aware of a similar problem that’s being looked into. If the Appium devs can reproduce the problem then it’s much easier to fix.

How do I make static text (e.g. labels) appear in Appium inspector?

Properly making your app accessible, such as adding accessibility labels, is well documented by Google and Apple for Android and iOS respectively. I suggest consulting the documentation. Once your app is properly following the guidelines, then the labels will be visible to appium for automation.

How transparent would it be to associate Android and iOS devices/VMs/Emulators with our Selenium Grid infrastructure to use automated tests using Appium?

It’s entirely possible to set up local infrastructure. In my experience, this involves a lot of work so it’s better to use a service provider such as Sauce Labs.

I have seen user reporting lost of issue in Maverics OS using Appium …. how does the Appium project plan to address all these issues? 

I have been using Appium on Mavericks without any issues. Make sure to use the newest version of OS X, Xcode, and Appium.

Is it possible to switch between native app and Safari when executing the workflow?

On iOS, this isn’t possible. The automation session is restricted to the app context and switching out will end the session. For Android, you could switch between the app and the Chrome browser without issue.

I use Firefox’s Responsive Web & UserAgent override to inspect Mobile Web Objects. Is there anything on the roadmap to inspect WebObjects directly from Appiums Inspector?

I recommend using the Chrome Dev tools instead of waiting for the Appium inspector to support web apps. The existing browser tools already work well for this purpose.

I’d love your recommendation for the best emulator for both iPhone and Android

This is highly specific to the business requirements of the app under test. I test on Android using a Nexus 7 and an iPhone Simulator. Sauce Labs supports a bunch of configurations to it’s up to the tester to decide what screen sizes and devices make sense.

I’m working on a mobile website project. Will Appium work for just mobile web, or is it specifically for hybrid and native apps?

Appium works well for mobile web, hybrid, and native apps. It’s not exclusive to any one of them.

If someone wanted to contribute to the most immediate need, what would that be? Features? Documentation? Quick Start Guide? Specific Language Usage Examples?

The most immediate need is documentation. The documentation files are hosted on GitHub and pull requests are welcome. Other areas to contribute include:

If we want to be able to fully automate testing, we need to be able to respond to system prompts, like those that appear to allow the app to use location services or access contacts. Our experience with Appium thus far is that there’s no means to respond. Can you elaborate?

I suggest asking this question on the discussion list. I’m pretty sure that someone has figured out a work around.

In the future, will Appium support running multiple mobile test cases on simulators/emulators and real devices in parallel, instead of running tests one after another? 

That’s already supported on Android. For iOS, Apple does not allow it. I suggest using a solution provider such as Sauce Labs that uses VMs to bypass this Apple limitation.

Inspector of Appium for Windows is much Flakey. Selendroid now doesnt gives Bounds/Size in PageSource.

For Selendroid, try their dedicated inspector tool

Is Appium 1.0.0 stable working for older Android like 4.2.x?
Appium works well on older Androids via Selendroid. uiautomator with Appium is available for API 17 and above.

Is Appium mainly for functional testing or integration testing? My company’s mobile device hits a webservice which returns a SQLite database, which we need to inspect for accuracy, once it’s been returned. Is this possible with Appium, even partially?

Appium is meant for end to end UI Testing, similar to how WebDriver is used to automate browsers. The situation described seems better suited for a web service unit test. I don’t see how a user of the app would inspect a SQLite database so appium doesn’t seem appropriate.

Is it always required that the Android app I’m testing have accessibility features (to locate the elements)? or I can use the simulator/recorder to locate those elements record tests for any app?

Accessibility features are not required. You can automate the app anyway however the other locator strategies will be brittle. The benefit of accessibility is the labels remain constant over the life cycle of the app. As the app changes, it’s better if tests continue working without having to update the locators.

Is there a plan to support testing on multiple iOS devices?

Apple hasn’t communicated about lifting this restriction. The current supported way is using Sauce Labs or another provider that handles parallelization.

Is there an easy way to setup Appium registered “devices” to Selenium GRID so that you can have a single point of access for tests?

A selenium grid is possible to setup. It’s not easy.

Just a suggestion, I also believe that we need more robust & rigorous UnitTest to be integrated with Appium CI, as the Appium release breaks a functionality or two while delivering another cool functionality.

I completely agree. I have experienced this issue and understand the frustration regressions cause. Proper unit testing with continuous integration is a work in progress. The goal is to have all Appium bindings in addition to the Appium server running robust unit tests via CI.

Last time I tried Appium, I had difficulty dealing with alerts coming from the OS or iTunes store (for payment automation). Does Appium 1.0 deal with OS/iTunes alerts better?

I’m not sure. If there’s a way to automate them with UI Automation JavaScript, then it’ll work with appium.

I need Appium/Node Logs to be consumed using a remote webdriver client. The WebDriver client is on remote m/c, so reading the log file doesn’t help much. Can we have a method to get log dumps via Appium-driver?

I believe there’s an open issue related to this. It makes sense to expose the logs via the Selenium log API. Currently supported logs include syslog (iOS), crashlog (iOS), and logcat (Android).

New to Appium, will soon use it. I briefly went over the multi action gestures. Are there any complex gestures that are not yet implemented? (or are still quirky/buggy)

So far the gestures seem to be working well.

One more question: Does Appium support C#, Java and Python languages?
Yes. Appium has client bindings for Ruby, Python, Java, JavaScript, PHP, and C#.

One of the big pain points for our QA is matching the implemented UI with the screen shots that our design team has put together. One guy in particular reached the laughing/crying stage imagining aloud the possibility of automation handling the comparison.

My personal opinion is that automatically matching design images to an app is not going to produce useful results. Appium supports screenshots so you’re welcome to try.

One of the requirements is XCode. Is Appium compatible with other IDEs, e.g. AppCode by Jetbrains?

Appium requires that Xcode is installed, not that you use it. Appium works with every IDE and can even be used without one.

Parallel execution always wonderful when it comes to faster testing. But iOS again some limitations. How are we going to address this?

The immediate solution is to use Sauce Labs. The longer term solution is to persuade Apple to stop limiting their automation technology.

Possibility to mock/stub webservices calls?

Mocks and Stubs are certainly possible. This is a general testing issue and is unrelated to appium.

Robotium can assert Android toasts by using something like waitForText(). Can Appium do the same thing?

uiautomator does not support toasts. Selendroid does.

A Selenium grid basically helps to redirect test cases to right devices … will this help with iOS as we are not able to run the iOS devices in parallel due to Apple limitation.

You could setup your own grid or use an existing solution provider such as Sauce Labs. Grid is not a magic solution to Apple’s limitation on parallelization.

Should we also depreciate/discourage usage of Android version less than 4.2?(Coz UIAutomator came at 4.2, and Selendroid is not stable to use)

I haven’t had good luck with Selendroid either so I only use uiautomator.

I’m attending this webinar to learn about Appium as someone who has never used it to automate testing. I enjoy being able to view screen shots in Sauce when I check my Selenium results… can Appium tests provide the same kind of screen shot results?

Yes, Sauce provides the same support for Appium including screenshots and videos. Locally, you would have to write the code to take the screenshots. I have open sourced a screen recorder for OS X that works with the iOS Simulator and Android emulator.

We had trouble writing tests that interact with tableViews, despite the fact that they are a common element  in iOS gui development. We’ve seen that this is on the list of known issues. Is addressing this on the roadmap in the near future?

I suggest opening an issue on GitHub and include an example that reproduces the problem. My understanding is that tableViews should not have problems on iOS.

Does Sauce Labs provide training for using Appium with Sauce? 

That’s a question to ask the sales team at Sauce Labs. I am working on screencast training that will be available to everyone for free.

Can you speak to whether/when you plan to address how to respond to prompts? Any workarounds available now?

I suggest posting on the discussion list. There are various ways to respond to prompts.

We had trouble migrating tests that ran on Firefox into tests that ran on Internet Explorer. Is it really that easy to migrate our tests to mobile with different browsers?

Appium enables you to automate mobile browsers. Chrome is a different browser from Safari so there will be differences. Browser automation has been successfully done for many years now. It’s entirely possible to test on multiple browsers.

We have the behavior that tests run locally stable and sometimes fail on the Appium server. For us this is not reproducible and could not determine why we have this instability. Do you know that? How do fix?

Flakiness can be caused by a large number of issues:

  • test written incorrectly
  • timing issues
  • app bugs
  • appium server bugs
  • appium client bugs
  • underlying automation bugs
  • emulator/simulator bugs
  • device specific bugs
  • network connectivity
  • environment configuration
  • software versions

If you post on the discussion list with specifics, then someone may be able to provide guidance.

We start our Appium tests with Jenkins and have in one example job 5 tests. From time to time we have 6 test result? Do you know why? It it a Appium issue?

This isn’t an Appium issue. There’s something broken with your test infrastructure if you’re randomly having an extra test result.

What are the changes app developers need to make in-app in order for visible property to be true in Appium inspector?

On Android, everything is always visible if it’s in the inspector. On iOS, if the element is displayed on screen then it should be visible -although sometimes it isn’t. Unfortunately there’s not much you can do to control the visibility attribute.

What are your future goals for automating  native apps for Samsung TV and Apple TV using Appium?

I’m not aware of specific goals to support TV automation. If the TV runs Android then it may work.

What is the best test reporting plugin for mobile that would fit with Appium (JUnit) to be included in CI?

JUnit has numerous plugins for various CI solutions that enable reporting. This isn’t Appium specific or unique to mobile. I suggest researching online to find out what others are using.

What is the main difference between Selandroid and Appium?

Selendroid only supports Android and uses an instrumentation approach. Appium allows automation of iOS and Android. Appium supports Selendroid as one option and also allows using the more modern uiautomator technology.

What native application support does Appium offer?

Appium supports fully automating native apps on Android and iOS. On Android there’s a choice of Selendroid or uiautomator. On iOS, UI Automation JavaScript is used behind the scenes.

Please suggest an operating system on which one should primarily test when launching Android – I.E. Linux, OSX, Windows…?

OS X

When I run Appium, I got the following error, “Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.” What can I do on my end to fix this?

This is a common issue. I suggest posting to the mailing list if it happens frequently.

When I tried Appium I found it was difficult to pick objects; it seems I need extensive xpath knowledge. Do you have plan to create an object library layer?

The Ruby binding has a helper library so you can use a generic find command that’ll work on almost anything. There are specific helpers for buttons, textfields, text, and alerts. I recommend not using XPath on Android as it’s flaky due to a known Android problem. With all the selector methods available, finding elements is very easy. First select which attribute you’re using to find the element then select the best selector.

When might Ruby users expect switch_to() to be implemented? We have many  hybrid apps that need context switching.

The new Ruby bindings have implemented context switching via set_context. Known issues are in the process of being fixed; these will improve the reliability of context switching.

Will there be a Appium conference soon?

App-solutely! For information about the Appium conference, please read this mailing list thread.

Will there ever be integrated support for CI though Sauce Labs?

Sauce Labs integrates with CI providers such as CloudBees. All CI solutions should be able to integrate with Sauce Labs. There are even plugins available.

You mentioned earlier that you are working on ‘Getting Started’ materials and tutorials… any idea on when you might be ready to release these and where should I look to find them?

These are already available on Appium.io. Check out the documentation, tutorials, and sample code.

You mentioned you now now support Android hybrid apps without Selenium. Which version of Appium supports this, and are there docs that show how to do this?

On Android API 19, Appium is able to connect directly to webviews via chromedriver when using uiautomator. The client libraries have support for listing the webview context and switching into it and back to the native app.

Guest Post: Bridging the Test Divide – Beyond Testing For A Release

June 16th, 2014 by Amber Kaplan

This is the second of a three part series by Matthew Heusser, software delivery consultant and writer. 

When I start to think about testing, I think about it in two broad strokes: new feature testing and release-testing. New feature testing tries to find problems with something new and specific, while release-testing happens after “code complete”, to make sure the whole system works together, that a change here didn’t break something there.

Release-testing (which some call regression testing) slows down the pace of release and delays feedback from our customer. Release-testing also increases cycle time – the time from when we begin work on a feature until it hits production. Over time, as our software become more complex, the amount of testing we want to do during release testing goes up.

Meanwhile, teams want to ship more often, to tighten the feedback loop.

Today I am going to talk about making release testing go away – or at least drastically reducing it.

It all starts during that tutorial in Spain I wrote about last time.

Two Worlds

The frequency of release for the people in my tutorial was very diverse, but two groups really struck me — the telecom that had a four-month test-release cycle, and the Latvian software team with the capability to deploy to production every single day.

That means arriving at the office the morning, looking at automated test runs, and making a decision to deploy.

There is a ‘formula’ to make this possible. It sounds simple and easy:

  • Automate a large number of checks on every build
  • Automate deploy to production
  • Continuously monitor traffic and logs for errors
  • Build the capability to rollback on failure

That transforms the role of test from doing the “testing we always do” to looking at the risk for a given release, lining it up against several different test strategies, and balancing risk, opportunity, reward, and time invested in release-testing?

The trick is to stop looking at the software as a big box, but instead to see it as a set of components. The classic set of components are large pieces of infrastructure (the configuration of the web server, the connections to the database, search, login, payment) and the things that sit on top of that – product reviews, comments, static html pages, and so on. Develop at least two de-ploy strategies — one for audited and mission-critical systems (essential infrastructure, etc) and another for components and add-ons.

We’ve been doing it for years in large IT organizations, where different systems have different release cycles; the trick is to split up existing systems, so you can recognize and make low-risk changes easier.

This isn’t something I dreamed up; both Zappos and Etsy have to pass PCI audits for financial services, while Zappos is part of Amazon and publicly traded. Both of these organizations have a sophisticated test-deploy process for parts of the application that touch money, and a simpler process for lower-risk changes.

So split off the system into different components that can be tested in isolation. Review the changes (perhaps down to the code level) to consider the impact of the change, and test the appropriate amount.

This can free up developers to make many tiny changes per day as long as those changes are low risk. Bigger changes along a theme can be batched together to save testing time — and might mean we can deploy with still considerably less testing than a ‘full’ site retest.

But How Do We Test It?

A few years ago, the ideal vision of getting away from manual, documented test cases was a single ‘test it’ button combined with a thumbs up or down at the end of an “automated test run.”

If the risk is different for each release, and we are uncomfortable with our automation, then we actually want to run different tests for each release — exactly what thinking testers (indeed, anyone on the team) can do with exploratory testing.

So let the computers provide some automated checks, all the time. Each morning, maybe every half an hour, we get a report, look at the changes, and decide what is the right thing for this re-lease. That might mean full-time exploratory testing of major features for a day or two, it might be emailing the team and asking everyone to spend a half hour testing in production.

This result is grown up software testing, varying the test approach to balance risk with cost.

The first step that I talked about today is separating components and developing a strategy that changes the test effort based on which parts were changed. If the risk is minimal, then deploy it every day. Hey, deploy it every hour.

This formula is not magic. Companies that try it find engineering challenges. The first build/deploy system they write tends to become hard to maintain over time. Done wrong continuous testing creates systematic and organizational risk.

It’s also a hard sell. So let’s talk about ways to change the system to shrink the release-test cycle, deploy more often, and reduce risk. The small improvements we make will stand on their own, not threaten anyway — and allow us to stop at any time and declare victory!

A Component Strategy

that_badWhen a company like etsy.com says that new programmers commit and push code to production the first day, do they really mean modifications to payment processing, search, or display for all products?

Of course not.

Instead, programmers follow a well-written set of directions to … wait for it … add the new user to the static HTML ‘about us’ page that lists all the employees, along with an image. If this change generates a bug, that will probably result in an X over an image the new hire forgot to upload, or maybe, at worst, break a div tag so the page mis-renders.

A bad commit on day one looks like this – not a bungled financial transaction in production.

How much testing should we have for that? Should we retest the whole site?

Let’s say we design the push to production so the ‘push’ only copies HTML and image files to the webserver. The server is never ‘down’, and serves complete pages. After the switch, the new page appears. Do we really need to give it the full monty, the week-long burn down of all that is good and right in testing? Couldn’t the developer try it on a local machine, push to stag-ing, try again, and “just push it?”

Questions on how?

More to come.

By Matthew Heusser – matt.heusser@gmail.com for Sauce Labs

Stay tuned next week for the third part of this mini series! You can follow Matt on Twitter at @mheusser.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

DevNetwork Recognizes Sauce Labs as a Leader in App Testing and Automation Technology [INFOGRAPHIC]

May 8th, 2014 by Amber Kaplan

Thanks to DevNetwork for including Sauce Labs in their infographic, “Leadership Map of Developer Technology” as a leader in App Testing and Automation Technology! You can see where Sauce fits into the ecosystem in the infographic below.

Dev Network

DevNetwork, producers of APIWorld, DataWeek, and DeveloperWeek, organized over 120 companies into 9 core groups of technologies across 34 categories in this version one edition of the developer technology industry.

“The emergence of a new ecosystem of cloud tools and technologies for developers means that almost every company no longer needs to feel locked into one technology stack or platform,” said Geoff Domoracki, CEO of DevNetwork.  “Because there are hundreds of technologies offering platform-as-a-service, infrastructure-as-a-service, database-as-a-service, even dev ops or back-end processing via API we saw the value to the community that we outline a comprehensive view of the best developer technologies in this infographic.”

About DevNetwork

DevNetwork is a San Francisco based technology conference company with over 6,500 participants across its conferences and events per year. These include APIWorld 2014, the nation’s largest API conference that’s 100% focused on developers, as well as DeveloperWeek 2015, which has grown to be San Francisco’s largest week-long tech event series. ContactDevNetwork to get your company involved in its media and events.

Ask a Selenium Expert: Selenium Grids, Scaling, and Parallelization

May 7th, 2014 by Amber Kaplan

selenium testing & sauceThis is part 5 of 8 in a mini series of follow-up Q&A’s from Selenium expert Dave Haeffner. Read up on the firstsecondthird, and fourth.

Dave discussed  how to build out a well factored, maintainable, resilient, and parallelized suite of tests that run locally, on a Continuous Integration system, and in the cloud in our recent webinar, “Selenium Bootcamp“.

Following the webinar, there were several follow-up questions. Dave’s agreed to respond to 8. Below you’ll find the fourth Q&A. Stay tuned next Wednesday for the next question.

5. ­Is Selenium Grid still relevant/useful for parallelization? ­

Selenium Grid is a great option for scaling your test infrastructure if you’re okay with handling the overhead of spinning up/maintaining a bunch of machines. It, by itself, will not give you parallelization. That is to say, it can handle however many connections you throw at it, but you will still need to find a way to execute your tests in parallel. You can learn more about Selenium Grid on it’s project main page.

-Dave Haeffner, April 9, 2014

Can’t wait to see the rest of the Q&A? Read the whole post here.  Get more info on Selenium with Dave’s book, The Selenium Guidebook, or follow him on Twitter or Github.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Ask a Selenium Expert: How to Run One Test on Multiple Browsers

April 30th, 2014 by Amber Kaplan

selenium testing & sauceThis is part 4 of 8 in a mini series of follow-up Q&A’s from Selenium expert Dave Haeffner. You can read up on the firstsecond, and third.

Dave discussed  how to build out a well factored, maintainable, resilient, and parallelized suite of tests that run locally, on a Continuous Integration system, and in the cloud in our recent webinar, “Selenium Bootcamp“.

Following the webinar, there were several follow-up questions. Dave’s agreed to respond to 8. Below you’ll find the fourth Q&A. Stay tuned next Wednesday for the next question.

4. ­How can I specify to Sauce Labs to run 1 test on multiple browsers?

You can see an example of one way to accomplish it in these write-ups:

-Dave Haeffner, April 9, 2014

Can’t wait to see the rest of the Q&A? Read the whole post here.  Get more info on Selenium with Dave’s book, The Selenium Guidebook, or follow him on Twitter or Github.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!) here.

Ask a Selenium Expert: Strategies for Cross-Browser Testing

April 23rd, 2014 by Amber Kaplan

selenium testing & sauceThis is part 3 of 8 in a mini series of follow-up Q&A’s from Selenium expert Dave Haeffner. Read up on the first and second

Dave discussed  how to build out a well factored, maintainable, resilient, and parallelized suite of tests that run locally, on a Continuous Integration system, and in the cloud in our recent webinar, “Selenium Bootcamp“.

Following the webinar, we had several follow-up questions. Dave’s agreed to respond to 8. Below you’ll find the third Q&A. Stay tuned next Wednesday for the next question.

3. ­I would like to see strategies for getting tests to work in multiple browers. For example, if my test works in Chrome but not Firefox, what do I do?

There are two things you’re likely to run into ­when running your tests across multiple browsers: speed of execution, and locator limitations.

Speed of execution issues occur when things execute too quickly (which indicates that you need to add explicit waits to your test code) or timeout (which indicates that the explicit waits you have are too tight and need to be loosened). The best approach to take is an iterative one. Run your tests and find the failures. Take each failed test, adjust your code as needed, and run it against the browsers you care about. Repeat until everything’s green.

In older browsers (e.g., Internet Explorer 8) you’ll be limited by the locators you can use (e.g., CSS3 nth locators like nth-child, nth-of-type, etc) and likely run into issues with some dynamic functionality (e.g., hovers). In cases like this, it’s simple enough to find an alternative set of locators to use that are specific to this browser and update your test to use them only when this browser is being used.

-Dave Haeffner, April 9, 2014

Can’t wait to see the rest of the Q&A? Read the whole post here.  Get more info on Selenium with Dave’s book, The Selenium Guidebook, or follow him on Twitter orGithub.

Have an idea for a blog post, webinar, or more? We want to hear from you! Submit topic ideas (or questions!)  here.

Bleacher Report Uses Appium by Sauce to Achieve “Olympic Quality” Results to Win at QA Testing (VIDEO)

February 28th, 2014 by Amber Kaplan

It’s an Olympic year, and Bleacher Report gets the gold when it comes to QA mobile testing. Winning!

Bleacher Report is a digital media company that delivers engaging content to sports fans all over the world. With more than half of their traffic coming from mobile devices, the team knew they needed to find a way to automatically test their mobile apps and mobile web experience. So they turned to Sauce Labs and Appium. Check out the video below to learn more.

Are you using Appium to test all your things? We’d love to hear what you’re working on. Leave us a comment on this post, tweet at us, or sign up for a free Sauce account to start testing.

Announcing iOS 7 support in the Sauce Labs Cloud

October 31st, 2013 by Ashley Wilson

ios7_01

Today we’re excited to let you know that you can now test your hybrid and native apps with iOS 7 on Sauce. With more than 200M downloads since its release, we know that this is an important platform for our users to test on.

To get started, visit our browsers and platforms page for Appium and copy and paste the DesiredCapabilities provided for either version of iOS (or Android) you want to test on. And if you’re new to mobile test automation in general, check out our Getting Started with Appium guide.

If you have any issues, give us a shout. Otherwise, happy (iOS 7) testing!