Calabash-ios on Windows

In the last post I discussed about setting up the Mac environment for running Calabash-ios tests from Xamarin studio. In Mac, we can run the calabash-tests on simulator easily by using the Fast Track steps if using Xcode or by using these steps if using Xamarin studio. To run on a device in mac we need to identify the device ip address  and use it in the terminal like this

DEVICE_ENDPOINT=http://192.168.1.146:37265 NO_LAUNCH=1 calabash-ios console

NO_LAUNCH=1 disables the simulator launch. Now if you launch the app in the device and run queries in the terminal it would work.

Running tests on Windows machine

On Windows machine we can run tests only on device and not on simulator(couldnt find a simulator for ios on windows yet).  You can setup Calabash framework on windows using gem install as specified in the calabash ios github wiki page.

Since we cannot build the Xamarin ios project completely on windows without a mac host, we assume that we have a .ipa file built from Mac setup which has the Calabash-framework force loaded. Copy the .ipa file built from Mac to Windows machine where you want to run tests. Connect the device to your windows machine and then open the .ipa file using itunes. It will list the app(say MyApp) under Apps category in itunes and then you can drag and copy it to the device(your device would be listed in the side pane and when you drag the app it will show the connected device and you can drop on the device name and it would install in device). Then you should be able to launch the app in the device and it would also have started the calabash-server. 

And , if you are following the cucumber.yml file(as mentioned in field-service-example), you can modify the ios profile like this,

ios: DEVICE_ENDPOINT=http://[ipofdevice]:37265 NO_LAUNCH=1 DEVICE=[udid] CALABASH_FULL_CONSOLE_OUTPUT=TRUE NO_STOP=1 OS=ios6 PLATFORM=ios -r features/support -r features/ios/support -r features/ios/helpers -r features/step_definitions -r features/ios/pages

and then in command prompt you can run something like(for login feature) cucumber -p ios features/login.feature .

NOTE: I haven't yet figured out how to use the bundle id and have the tests automatically start the app in device yet and if i do so, I will either update this post or enter a new blog to keep track of it.

Happy Testing :)

Android Automation Testing

After evaluating couple of automation testing platforms(like Robotium, Ranorex etc), I found Calabash  the simplest platform that supports cross platform mobile application automation testing. Another requirement I had was to have the ability to test applications developed using the Mono for Android/ IOS frameworks.  Calabash comes in two flavors Calabash-Android and Calabash-IOS and it also had the option to test in the cloud environment through less-painful which apparently had been acquired by Xamarin.  This post covers only the details of setting up the test environment for Android apps.

Calabash-Android

Calabash supports behavior driven test case development using Cucumber. While the high level test case can be written using predefined step definitions in Cucumber, more complex steps can be defined in Ruby. Calabash-Android provides lot of predefined steps that supports lot of input steps, gestures, touch, snapshot etc. 

Installation

1. The first step is to setup Ruby. At the time of writing this post(July 1st 2013), Ruby 1.9.3 is the version supported by Calabash-Android. When Installing Ruby make sure the installed directory is set in the environment path(the last step in the installer prompts for this)
2. After installing Ruby 1.9.3,  install the Development kit.  Download the correct version of development kit as defined in the Ruby download page. For Ruby 1.8.6 to 1.9.3,  tdm-32-4.5.2 is the suggested devkit.
3. Follow the detailed instructions as given here to setup the devkit. Its basically these steps
• Extracting the Devkit installer(7-zip archive) to a <Install> directory.
• In the Cmd prompt, enter Cd <Install> , where <Install> is the extracted directory in the previous step
• Then enter, ruby dk.rb init . This would generate the config.yml file which has details about the installed ruby packages on the system. You can verify that the  Ruby 1.9.3 is present in this file
•  Then enter ruby dk.rb install to finalize the devkit installation
4. The next step is to install the Calabash-android gem. Enter gem install calabash-android in the cmd prompt.

Testing the App

Now that the Calabash-android framework is installed and setup the next step is to get the App ready to test. Calabash-Android provides a command utility to easily set up the test project for the app.

In the cmd prompt, enter the directory in which you want to store the test cases. Let this be the <TestDir>.  

• Enter cd <TestDir> in the Cmd prompt. 
• Then type calabash-android gen  to generate the stub files for the test cases. it would generate the files in this format

features

|_support
| |_app_installation_hooks.rb
| |_app_life_cycle_hooks.rb
| |_env.rb
|_step_definitions
| |_calabash_steps.rb
|_my_first.feature

The .feature file will have the cucumber style test case like this

Feature: Login feature

  Scenario: As a valid user I can log into my app
    When I press "Login"
    Then I see "Welcome to MyApp"

Assuming your app has the welcome screen with Login button and the welcome text,        running the login feature test case would succeed without any changes. But typically you 

would want to change the login feature file by providing valid login credentials and ensuring that we see the success message. You can achieve this with the Predefined Steps defined by   the calabash framework. Assuming you have gone through the calabash android framework fair enough and wrote the required steps to validate the login scenario of your app,  you have to sign the .apk file(say com.fun.myapp.apk) with the debug key to test the app.  

The following steps need to be followed to run the test

To help the setup process easier, set the Java SDK path and the Android SDK path in the JAVA_HOME and ANDROID_HOME variables. 
type the following cmd,

cd %JAVA_HOME%\bin
jarsigner.exe -verbose -sigalg MD5withRSA -digestalg SHA1 -keystore debug.keystore -storepass android -keypass android %~dp0\com.fun.myapp.apk androiddebugkey

  3. Allign the .apk file with the following comman

cd %ANDROID_HOME%\tools\
zipalign -v 4 %~dp0\com.fun.myapp.apk %~dp0\com.fun.myapp_allign.apk 

 4. Run the tests and export the report to .html file and also logging the output in the console

calabash-android run com.fun.myapp_allign.apk --format html --out report.html --format pretty

You can combine all these steps in a batch to make it convenient to run each time. here is a snapshot of the batch file

Now you should be able to see the output in either the report.html file or in the console. Their report format is really cool and it shows the list of features run and what succeeded and failed with color highlights.