Android Automation Testing

After evaluating couple of automation testing platforms(like Robotium, Ranorex etc), I found Calabash  the simplest platform that supports cross platform mobile application automation testing. Another requirement I had was to have the ability to test applications developed using the Mono for Android/ IOS frameworks.  Calabash comes in two flavors Calabash-Android and Calabash-IOS and it also had the option to test in the cloud environment through less-painful which apparently had been acquired by Xamarin.  This post covers only the details of setting up the test environment for Android apps.

Calabash-Android

Calabash supports behavior driven test case development using Cucumber. While the high level test case can be written using predefined step definitions in Cucumber, more complex steps can be defined in Ruby. Calabash-Android provides lot of predefined steps that supports lot of input steps, gestures, touch, snapshot etc. 

Installation

1. The first step is to setup Ruby. At the time of writing this post(July 1st 2013), Ruby 1.9.3 is the version supported by Calabash-Android. When Installing Ruby make sure the installed directory is set in the environment path(the last step in the installer prompts for this)
2. After installing Ruby 1.9.3,  install the Development kit.  Download the correct version of development kit as defined in the Ruby download page. For Ruby 1.8.6 to 1.9.3,  tdm-32-4.5.2 is the suggested devkit.
3. Follow the detailed instructions as given here to setup the devkit. Its basically these steps
• Extracting the Devkit installer(7-zip archive) to a <Install> directory.
• In the Cmd prompt, enter Cd <Install> , where <Install> is the extracted directory in the previous step
• Then enter, ruby dk.rb init . This would generate the config.yml file which has details about the installed ruby packages on the system. You can verify that the  Ruby 1.9.3 is present in this file
•  Then enter ruby dk.rb install to finalize the devkit installation
4. The next step is to install the Calabash-android gem. Enter gem install calabash-android in the cmd prompt.

Testing the App

Now that the Calabash-android framework is installed and setup the next step is to get the App ready to test. Calabash-Android provides a command utility to easily set up the test project for the app.

In the cmd prompt, enter the directory in which you want to store the test cases. Let this be the <TestDir>.  

• Enter cd <TestDir> in the Cmd prompt. 
• Then type calabash-android gen  to generate the stub files for the test cases. it would generate the files in this format

features

|_support
| |_app_installation_hooks.rb
| |_app_life_cycle_hooks.rb
| |_env.rb
|_step_definitions
| |_calabash_steps.rb
|_my_first.feature

The .feature file will have the cucumber style test case like this

Feature: Login feature

  Scenario: As a valid user I can log into my app
    When I press "Login"
    Then I see "Welcome to MyApp"

Assuming your app has the welcome screen with Login button and the welcome text,        running the login feature test case would succeed without any changes. But typically you 

would want to change the login feature file by providing valid login credentials and ensuring that we see the success message. You can achieve this with the Predefined Steps defined by   the calabash framework. Assuming you have gone through the calabash android framework fair enough and wrote the required steps to validate the login scenario of your app,  you have to sign the .apk file(say com.fun.myapp.apk) with the debug key to test the app.  

The following steps need to be followed to run the test

To help the setup process easier, set the Java SDK path and the Android SDK path in the JAVA_HOME and ANDROID_HOME variables. 
type the following cmd,

cd %JAVA_HOME%\bin
jarsigner.exe -verbose -sigalg MD5withRSA -digestalg SHA1 -keystore debug.keystore -storepass android -keypass android %~dp0\com.fun.myapp.apk androiddebugkey

  3. Allign the .apk file with the following comman

cd %ANDROID_HOME%\tools\
zipalign -v 4 %~dp0\com.fun.myapp.apk %~dp0\com.fun.myapp_allign.apk 

 4. Run the tests and export the report to .html file and also logging the output in the console

calabash-android run com.fun.myapp_allign.apk --format html --out report.html --format pretty

You can combine all these steps in a batch to make it convenient to run each time. here is a snapshot of the batch file

Now you should be able to see the output in either the report.html file or in the console. Their report format is really cool and it shows the list of features run and what succeeded and failed with color highlights.