How risky is the testing of mobile application only on simulator / emulator

Launching the application based on the testing on emulators / simulators could be very risky for companies / developers and may affect the brand value of the application and company. Developing a mobile application is very easy and quick these days and requires very minimum investment. Now a days with the existence of so many application stores, it is very easy to launch the application and reach to wide user base but if application has issues then it'll get negative feedback even if concept of the application is very good and has the potential of becoming top rated app. This happens mostly because of absence of proper testing of the application from the technical as well as user's point.

Many companies especially start-ups, do not invest much on testing of their mobile application that affects them adversely. Some companies do perform some testing but only on emulators / simulators instead of actual mobile handsets. It is very difficult to test the application on each and every available mobile handset but app should be tested on some candidate handsets. First few iterations of testing should be done on emulators / simulators but final testing should be done on a set of actual handsets. Following are some cons of testing application only on emulators / simulators:

1. Network issue: Emulators emulate the network speed but emulation is not 100% correct and most of the emulators use the network speed of the PC so performance of AUT on emulator / simulator may not be very reliable.
2. Resolution issue: Screen size and resolution of a PC also affects the app UI especially in case of gaming apps e.g. user experience on 4 inch screen having 1136x640 resolution is different from that on 4 inch screen having 800x480 resolution i.e user experience changes based on pixel density. Emulating exact screen size and screen resolution is very difficult on emulators as emulators use resolution of the PC.
3. PC resources: Emulator uses PC resources because of which actual user experience may be very different from that on emulators.
4. Apps using device camera or GPS cannot be tested properly on emulators.
5. Carrier specific testing cannot be done on emulators.
6. It is difficult to do interrupt testing (e.g. Call / SMS etc.) on emulators.

Testers are no more road blocker

Few days back I attended a conference on "Verification Futures". It was primarily for hardware verification engineers. There were speakers from major EDA vendors like Synopsys, Mentor and Cadence. They talked about current challenges faced by verification engineers and innovations happening in the industry.

A key trend which is observed, is change of focus from design oriented VIPs to verification oriented VIPs which means if anything cannot be tested and then it should not be designed which is very true in software industry also.

As in software industry, focus is moving from first develop and then test to first write test and then develop code for it i.e. TDD (Test Driven Development). As complexity of the softwares is increasing day by day so proper testing of the software is becoming very-very important and now both clients and senior management have realized the importance of testing and hence are more concerned about proper and thorough testing and testing artifacts. Now a days, testers are not just black box tester they dig into code to find probable issues and suggest solutions. Instead of performing only random testing, they are now more systematic and focused, they use both scientific methods as well as their experience. They are expected to have more technical as well as domain knowledge.

In essence now a tester is no more considered as someone who blocks the release of the product but as one who is integral part of the team and who provides confidence in the product.

Tester surprises everyone

"Oh this is also the functionality, I didn't know", have you ever heard this line from developers, probably yes. Many times we find a scenario which even developers are not aware of. This happens because each developer works on his/her module and may not be aware of the big picture of the product. Many times even product owner gets surprised with the issues which testers find in testing, because even he/she might not have thought from that point.

Product owner usually defines high level requirements so, it is the responsibility of the requirement gathering team to ask correct questions to get details. Many requirements are very obvious for PO since he/she is very familiar with the product domain, but those small and obvious requirements might play very important role in development and testing of the product.

PO looks at the product from user's point of view and a developer looks at the product from technical point of view. But a tester actually looks at the product from both user's point of view as well as technical point of view, so it is very important for him to be part of requirement gathering. Moreover everyone (developer / TL / PM / PO) look at the tester as a domain expert. So it is very important for tester to get some domain knowledge before actually starting testing. He has to think out of the box to find out some very important and severe issues in the product.

How to fire event from AUT to other app using robotium / instrumentation

IMPORTANT: Code will work only on rooted device or emulator.

Many times we are required to test an andorid app which talks to any other app like (messaging, web browser etc.) As robotium or android intrumention does not allow us to access other apps from within test code we end up testing that feature manually which is a big obstacle in automating AUT.

While trying to automate one of the android app which uses messaging, email and other features of device, we also got stuck with this problem. We googled a lot and thankfully we found some solution to our problem which we think of worth sharing with all.

First approach:
The first and simplest approach is using adb shell command to perform required operation on device/emulator. Here is the sample command where running adb command to come back from third party app.
                   Runtime.getRuntime().exec("adb shell input keyevent 4");

Here 4 is the keycode for Back key.
This approach is very useful when you have only apk file and you are not able to modify manifest file of AUT.

Second approach:
Second approach is using INJECT_EVENTS permission in the manifest file of AUT. But problem with this approach is you need to put your AUT in sys app folder of emulator/device and for which you need system certificate. If you do not have system certificate then you need to root your device or run this test on emulator. Steps are mentioned below:

1.  Add INJECT_EVENTS permission tag in the manifest file of AUT.
2.  Build AUT with android debug certificate and get apk file. 
3.  Download RootExplorer.apk or similar app from internet and install it on desired emulator/device. 
4.  Copy apk file of AUT to the emulator/device (DO NOT install yet), preferrably on sdcard. e.g. adb push AUT.apk /sdcard/ 
5. Open RootExplorer and go to /sdcard and copy apk file of AUT. 
6. Make sure current mode is R/W. To switch to R/W mode there is a button provided at the top left corner on RootExplorer screen.  
7. Navigate to /system/apps folder and paste apk file of AUT. 
8. Install AUT by tapping on it 
9. Now from your code you can fire any event when AUT interacts with other external activity e.g. If AUT has a button and clicking on it launches web browser then to navigate back from web browser to AUT you can use 

                 Instrumentation inst = getInstrumentation();
                 inst.sendKeyDownUpSync(KeyEvent.KEYCODE_BACK); 

 This approach gives us more flexibility and control over other app and is very useful if we have code of AUT.

Testing checklist for Android app

1. Core functionality of the app is working as expected.

2. UI of the app is as expected like all elements are aligned properly; there is no truncation of element or text etc. UI should be consistent across all screens of AUT

3. Touch area of all touchable elements is big enough so that normal user does not feel any problem in touching them.

4. Font size of all the text should be optimum i.e. neither too small nor too big so that normal user is able to read it with ease.

5. UI remains consistent on rotating the device. If user is reading some text then on rotating device text/list should not scroll to top i.e. it should maintain current position.

6. If on rotating device there is some extra space then elements should be automatically adjusted accordingly i.e. either size of the elements should be increased or other elements (for which user was required to scroll earlier) should be displayed on the screen.

7. Smart phones automatically handle most of the suspend resume functionality automatically but testers should also test it especially in case of audio & video play.

8. If AUT is accessing any third party / inbuilt app like contacts, messaging, email, facebook etc. then it should transfer required data properly and on coming back to AUT should be smooth and should come back to the same screen from where user left.

9. If AUT needs to download very large data then it is recommended to be downloaded asynchronously i.e. initial chunks of data should be downloaded first and user should be able to read text or scroll list while rest of the data is being downloaded.

10. If AUT has some authentication screen like login screen then keep an eye on network log to make sure that AUT does not pass username & password in plain text. Also after logout such authentication tokens / session id should be deleted.

11. If AUT is storing some personal information like authentication token, password, credit card then it should be saved in encrypted form.

12. DDMS should be used to check memory leaks and memory consumed by AUT during run.

13.  App should support all the orientation of the device. If not, then both variations of the orientation (in case of portrait mode - upside-down).

14. On uninstallation, all the files related to the AUT should get deleted and other apps should remain unaffected.

15. All navigation should be self-explanatory like Back button should take user to logically previous page, Cancel button should cancel current operation etc.

16. If AUT has scrollable form like screen then currently focused text field should be visible to user i.e. keyboard should not hide it. And on hiding keyboard screen should be adjusted accordingly.

17. Navigation from one screen to another or screen rotation should not be abrupt. AUT should animate components smoothly.

18. It is recommended that AUT should save all large data files on sd card not in phone memory.

19. If app is storing data on SD card then scenarios like SD not available, SD card removed in the middle of data transfer etc. should be tested properly.

20. All the required permissions should be defined in manifest file and it should not have permissions which are not required by AUT.

21. If AUT is a service which gets activated on certain events then those firing of those events should be tested properly. Also if AUT is required to be running in background continuously then scenarios like device reboot, user stops service explicitly should be tested.

22. Switching between Wi-Fi and mobile network should not affect functionality of AUT. Also if device’s Wi-Fi and network are disabled by user and AUT tries to make any network call then AUT should be display appropriate error message to user, asking him to enable Wi-Fi. In such scenario AUT should not display loading bar before should error message.

23. AUT should handle network failure scenario gracefully e.g. device loses network while AUT is downloading some content. In such scenario a proper error message should be displayed after timeout i.e. AUT should not keep displaying loading screen indefinitely.

24. Battery consumption should be under control when AUT is running.

25. Testing AUT using different network connectivity like Wi-Fi, 3G, 2G etc.

26. Special testing should be done to verify how AUT handles foreign characters and special characters.

27. If new version of AUT is available then user should be notified about the update and AUT should be updated smoothly without deleting existing user’s data and preferences.

28. One of the most important aspects is performance of the AUT. AUT should not be very slow otherwise user will move to other similar apps. AUT should keep user engaged and should display results as quickly as possible.

A very good android testing tree could be found here

Running android automation script using batch file

Run android automation script and copy test result files on local machine (using batch file):

Here, I am taking example of retrieving screenshots taken by robotium. Robotium test script, in this example, runs on emulator/device and takes screenshots at regular intervals. Screenshots taken by robotium are stored in sdcard/Robotium-Screenshots/ folder. For demo purpose, I am running script on emulator but same script can be run on device also.

Note: You can retrieve any file like test results etc. in the same manner. You can even first push test data files on device/emulator and after execution pull the result files.

To run on actual device:

1. Comment/Remove commands which launches emulator and other emulator related commands.
2. Make sure device is attached to system and adb devices show the device.

Create a batch file and copy paste following code

@ echo OFF
REM Path of android sdk
SET SDK_DIR=c:\android-sdk

REM Path of folder where screenshots will be copied
SET SCREENSHOT_DIR=c:\temp\screenshot
SET TIMESTAMP=%date:~-4,4%%date:~-10,2%%date:~-7,2%%time:~-11,2%%time:~-8,2%
SET APK_PACKAGE=%1
SET APK_FILE=%2
SET EMULATOR=%3
SET EMULATOR_PORT=%4

IF [%1]==[] GOTO ErrLbl
IF [%2]==[] SET APK_FILE=*.apk
IF [%2]==[] SET EMULATOR=testEmu
IF [%3]==[] SET EMULATOR_PORT=5554

echo Setting path environment variable
SET PATH=%PATH%;%SDK_DIR%\tools;%SDK_DIR%\platform-tools

SET SCREENSHOT_DIR=%SCREENSHOT_DIR%\%EMULATOR%_%EMULATOR_PORT%_%TIMESTAMP%
echo Creating the screenshot folder %SCREENSHOT_DIR%
mkdir %SCREENSHOT_DIR%

echo Launching emulator %EMULATOR%
emulator @%EMULATOR% -port %EMULATOR_PORT% -scale 0.8 -no-boot-anim –no-window
wait 120

echo Waiting for device to come online
adb wait-for-device

echo Press any key only when emulator is completely online.
pause
adb shell input keyevent 82

echo Installing apk file: %APK_FILE%
adb install %APK_FILE%

echo Deleting existing screenshots from device
adb shell rm sdcard/Robotium-Screenshots/*
pause

echo Running test: %APK_PACKAGE%
adb shell am instrument -w
%APK_PACKAGE%/android.test.InstrumentationTestRunner

echo Copying Screeshots from sdcard to local Dir - %SCREENSHOT_DIR%
adb  pull
sdcard/Robotium-Screenshots/ %SCREENSHOT_DIR%\

echo Deleting screenshots from device
adb shell  rm
sdcard/Robotium-Screenshots/*
pause
echo Killing emulator
REM This command does not work on Windows OS but should work on other OS
REM adb -s emulator-%EMULATOR_PORT% emu kill
REM For Windows here is the workaround which is not good solution but does desired task
taskkill /IM emulator-arm.exe
REM Below command is required to avoid execution of ErrLbl content
exit /B

:ErrLbl
echo Package name is MISSING.

Testing checklist for iOS app

1. Core functionality of the app is working as expected.

2. UI of the app is as expected like all elements are aligned properly; there is no truncation of element or text etc. UI should be consistent across all screens of AUT.

3. Touch area of all touchable elements is big enough so that normal user does not feel any problem in touching them.

4. Font size of all the text should be optimum i.e. neither too small nor too big so that normal user is able to read it with ease.

5. UI remains consistent on rotating the device. If user is reading some text then on rotating device text/list should not scroll to top i.e. it should maintain current position.

6. If on rotating device there is some extra space then elements should be automatically adjusted accordingly i.e. either size of the elements should be increased or other elements (for which user was required to scroll earlier) should be displayed on the screen.

7. Smart phones automatically handle most of the suspend resume functionality automatically but testers should also test it especially in case of audio & video play.

8. If AUT is accessing any third party / inbuilt app like contacts, messaging, email, facebook etc. then it should transfer required data properly and on coming back to AUT should be smooth and should come back to the same screen from where user left.

9. If AUT needs to download very large data then it is recommended to be downloaded asynchronously i.e. initial chunks of data should be downloaded first and user should be able to read text or scroll list while rest of the data is being downloaded.

10. If AUT has some authentication screen like login screen then keep an eye on network log to make sure that AUT does not pass username & password in plain text. Also after logout such authentication tokens / session id should be deleted.

11. If AUT is saving some personal information like authentication token, password, credit card then it should be saved in encrypted form.

12. Instrumentation should be used to check memory leaks and memory consumed by AUT during run.

13.  App should support all the orientation of the device. If not, then both variations of the orientation (in case of portrait mode - upside-down).

14. On uninstalling, all the files related to the AUT should get deleted and other apps should remain unaffected.

15. If AUT displays badge on the app icon then it should display correct count and on every update it should be updated accordingly.

16. If AUT has popover then popover should not have any explicit close button. Instead it should disappear when user taps on any item on it and corresponding action should take place or user tab anywhere else on screen.

17. All navigation should be self-explanatory like Back button should take user to logically previous page, Cancel button should cancel current operation etc.

18. If AUT has scroll-able form like screen then currently focused text field should be visible to user i.e. keyboard should not hide it. And on hiding keyboard screen should be adjusted accordingly.

19. Navigation from one screen to another or screen rotation should not be abrupt. AUT should animate components smoothly.

20. Switching between Wi-Fi and mobile network should not affect functionality of AUT. Also if device’s Wi-Fi and network are disabled by user (i.e. Airplane mode) and AUT tries to make any network call then AUT should be display appropriate error message to user, asking him to enable Wi-Fi. In such scenario AUT should not display loading bar before should error message.

21. Battery consumption should be under control when AUT is running.

22. One of the most important aspects is performance of the AUT. AUT should not be very slow otherwise user will move to other similar apps. AUT should keep user engaged and should display results as quickly as possible.

23. Testing AUT using different network connectivity like Wi-Fi, 3G, 2G etc.

24. AUT should handle network failure scenario gracefully e.g. device loses network while AUT is downloading some content. In such scenario a proper error message should be displayed after timeout i.e. AUT should not keep displaying loading screen indefinitely.

25. Special testing should be done to verify how AUT handles foreign characters and special characters.

26. If new version of AUT is available then user should be notified about the update and AUT should be updated smoothly without deleting existing user’s data and preferences.

For more details refer to Apple’s Human Interface Guidelines.

A very good iOS testing tree could be found here