Testing checklist for Android app

1. Core functionality of the app is working as expected.

2. UI of the app is as expected like all elements are aligned properly; there is no truncation of element or text etc. UI should be consistent across all screens of AUT

3. Touch area of all touchable elements is big enough so that normal user does not feel any problem in touching them.

4. Font size of all the text should be optimum i.e. neither too small nor too big so that normal user is able to read it with ease.

5. UI remains consistent on rotating the device. If user is reading some text then on rotating device text/list should not scroll to top i.e. it should maintain current position.

6. If on rotating device there is some extra space then elements should be automatically adjusted accordingly i.e. either size of the elements should be increased or other elements (for which user was required to scroll earlier) should be displayed on the screen.

7. Smart phones automatically handle most of the suspend resume functionality automatically but testers should also test it especially in case of audio & video play.

8. If AUT is accessing any third party / inbuilt app like contacts, messaging, email, facebook etc. then it should transfer required data properly and on coming back to AUT should be smooth and should come back to the same screen from where user left.

9. If AUT needs to download very large data then it is recommended to be downloaded asynchronously i.e. initial chunks of data should be downloaded first and user should be able to read text or scroll list while rest of the data is being downloaded.

10. If AUT has some authentication screen like login screen then keep an eye on network log to make sure that AUT does not pass username & password in plain text. Also after logout such authentication tokens / session id should be deleted.

11. If AUT is storing some personal information like authentication token, password, credit card then it should be saved in encrypted form.

12. DDMS should be used to check memory leaks and memory consumed by AUT during run.

13.  App should support all the orientation of the device. If not, then both variations of the orientation (in case of portrait mode - upside-down).

14. On uninstallation, all the files related to the AUT should get deleted and other apps should remain unaffected.

15. All navigation should be self-explanatory like Back button should take user to logically previous page, Cancel button should cancel current operation etc.

16. If AUT has scrollable form like screen then currently focused text field should be visible to user i.e. keyboard should not hide it. And on hiding keyboard screen should be adjusted accordingly.

17. Navigation from one screen to another or screen rotation should not be abrupt. AUT should animate components smoothly.

18. It is recommended that AUT should save all large data files on sd card not in phone memory.

19. If app is storing data on SD card then scenarios like SD not available, SD card removed in the middle of data transfer etc. should be tested properly.

20. All the required permissions should be defined in manifest file and it should not have permissions which are not required by AUT.

21. If AUT is a service which gets activated on certain events then those firing of those events should be tested properly. Also if AUT is required to be running in background continuously then scenarios like device reboot, user stops service explicitly should be tested.

22. Switching between Wi-Fi and mobile network should not affect functionality of AUT. Also if device’s Wi-Fi and network are disabled by user and AUT tries to make any network call then AUT should be display appropriate error message to user, asking him to enable Wi-Fi. In such scenario AUT should not display loading bar before should error message.

23. AUT should handle network failure scenario gracefully e.g. device loses network while AUT is downloading some content. In such scenario a proper error message should be displayed after timeout i.e. AUT should not keep displaying loading screen indefinitely.

24. Battery consumption should be under control when AUT is running.

25. Testing AUT using different network connectivity like Wi-Fi, 3G, 2G etc.

26. Special testing should be done to verify how AUT handles foreign characters and special characters.

27. If new version of AUT is available then user should be notified about the update and AUT should be updated smoothly without deleting existing user’s data and preferences.

28. One of the most important aspects is performance of the AUT. AUT should not be very slow otherwise user will move to other similar apps. AUT should keep user engaged and should display results as quickly as possible.

A very good android testing tree could be found here

Running android automation script using batch file

Run android automation script and copy test result files on local machine (using batch file):

Here, I am taking example of retrieving screenshots taken by robotium. Robotium test script, in this example, runs on emulator/device and takes screenshots at regular intervals. Screenshots taken by robotium are stored in sdcard/Robotium-Screenshots/ folder. For demo purpose, I am running script on emulator but same script can be run on device also.

Note: You can retrieve any file like test results etc. in the same manner. You can even first push test data files on device/emulator and after execution pull the result files.

To run on actual device:

1. Comment/Remove commands which launches emulator and other emulator related commands.
2. Make sure device is attached to system and adb devices show the device.

Create a batch file and copy paste following code

@ echo OFF
REM Path of android sdk
SET SDK_DIR=c:\android-sdk

REM Path of folder where screenshots will be copied
SET SCREENSHOT_DIR=c:\temp\screenshot
SET TIMESTAMP=%date:~-4,4%%date:~-10,2%%date:~-7,2%%time:~-11,2%%time:~-8,2%
SET APK_PACKAGE=%1
SET APK_FILE=%2
SET EMULATOR=%3
SET EMULATOR_PORT=%4

IF [%1]==[] GOTO ErrLbl
IF [%2]==[] SET APK_FILE=*.apk
IF [%2]==[] SET EMULATOR=testEmu
IF [%3]==[] SET EMULATOR_PORT=5554

echo Setting path environment variable
SET PATH=%PATH%;%SDK_DIR%\tools;%SDK_DIR%\platform-tools

SET SCREENSHOT_DIR=%SCREENSHOT_DIR%\%EMULATOR%_%EMULATOR_PORT%_%TIMESTAMP%
echo Creating the screenshot folder %SCREENSHOT_DIR%
mkdir %SCREENSHOT_DIR%

echo Launching emulator %EMULATOR%
emulator @%EMULATOR% -port %EMULATOR_PORT% -scale 0.8 -no-boot-anim –no-window
wait 120

echo Waiting for device to come online
adb wait-for-device

echo Press any key only when emulator is completely online.
pause
adb shell input keyevent 82

echo Installing apk file: %APK_FILE%
adb install %APK_FILE%

echo Deleting existing screenshots from device
adb shell rm sdcard/Robotium-Screenshots/*
pause

echo Running test: %APK_PACKAGE%
adb shell am instrument -w
%APK_PACKAGE%/android.test.InstrumentationTestRunner

echo Copying Screeshots from sdcard to local Dir - %SCREENSHOT_DIR%
adb  pull
sdcard/Robotium-Screenshots/ %SCREENSHOT_DIR%\

echo Deleting screenshots from device
adb shell  rm
sdcard/Robotium-Screenshots/*
pause
echo Killing emulator
REM This command does not work on Windows OS but should work on other OS
REM adb -s emulator-%EMULATOR_PORT% emu kill
REM For Windows here is the workaround which is not good solution but does desired task
taskkill /IM emulator-arm.exe
REM Below command is required to avoid execution of ErrLbl content
exit /B

:ErrLbl
echo Package name is MISSING.

Testing checklist for iOS app

1. Core functionality of the app is working as expected.

2. UI of the app is as expected like all elements are aligned properly; there is no truncation of element or text etc. UI should be consistent across all screens of AUT.

3. Touch area of all touchable elements is big enough so that normal user does not feel any problem in touching them.

4. Font size of all the text should be optimum i.e. neither too small nor too big so that normal user is able to read it with ease.

5. UI remains consistent on rotating the device. If user is reading some text then on rotating device text/list should not scroll to top i.e. it should maintain current position.

6. If on rotating device there is some extra space then elements should be automatically adjusted accordingly i.e. either size of the elements should be increased or other elements (for which user was required to scroll earlier) should be displayed on the screen.

7. Smart phones automatically handle most of the suspend resume functionality automatically but testers should also test it especially in case of audio & video play.

8. If AUT is accessing any third party / inbuilt app like contacts, messaging, email, facebook etc. then it should transfer required data properly and on coming back to AUT should be smooth and should come back to the same screen from where user left.

9. If AUT needs to download very large data then it is recommended to be downloaded asynchronously i.e. initial chunks of data should be downloaded first and user should be able to read text or scroll list while rest of the data is being downloaded.

10. If AUT has some authentication screen like login screen then keep an eye on network log to make sure that AUT does not pass username & password in plain text. Also after logout such authentication tokens / session id should be deleted.

11. If AUT is saving some personal information like authentication token, password, credit card then it should be saved in encrypted form.

12. Instrumentation should be used to check memory leaks and memory consumed by AUT during run.

13.  App should support all the orientation of the device. If not, then both variations of the orientation (in case of portrait mode - upside-down).

14. On uninstalling, all the files related to the AUT should get deleted and other apps should remain unaffected.

15. If AUT displays badge on the app icon then it should display correct count and on every update it should be updated accordingly.

16. If AUT has popover then popover should not have any explicit close button. Instead it should disappear when user taps on any item on it and corresponding action should take place or user tab anywhere else on screen.

17. All navigation should be self-explanatory like Back button should take user to logically previous page, Cancel button should cancel current operation etc.

18. If AUT has scroll-able form like screen then currently focused text field should be visible to user i.e. keyboard should not hide it. And on hiding keyboard screen should be adjusted accordingly.

19. Navigation from one screen to another or screen rotation should not be abrupt. AUT should animate components smoothly.

20. Switching between Wi-Fi and mobile network should not affect functionality of AUT. Also if device’s Wi-Fi and network are disabled by user (i.e. Airplane mode) and AUT tries to make any network call then AUT should be display appropriate error message to user, asking him to enable Wi-Fi. In such scenario AUT should not display loading bar before should error message.

21. Battery consumption should be under control when AUT is running.

22. One of the most important aspects is performance of the AUT. AUT should not be very slow otherwise user will move to other similar apps. AUT should keep user engaged and should display results as quickly as possible.

23. Testing AUT using different network connectivity like Wi-Fi, 3G, 2G etc.

24. AUT should handle network failure scenario gracefully e.g. device loses network while AUT is downloading some content. In such scenario a proper error message should be displayed after timeout i.e. AUT should not keep displaying loading screen indefinitely.

25. Special testing should be done to verify how AUT handles foreign characters and special characters.

26. If new version of AUT is available then user should be notified about the update and AUT should be updated smoothly without deleting existing user’s data and preferences.

For more details refer to Apple’s Human Interface Guidelines.

A very good iOS testing tree could be found here

Test WebView used within android native app

Unfortunately currently robotium does not support testing of webview even if it is used with native app but there is a way to test webview using Android WebDriver. So the solution is using combination of robotium + android webdriver. Here is the sample code to test contents of webview.
Little background about AUT: Native app has a button (assume ‘Google’) clicking on which opens webview with google.com.

Sample Code:
Activity mActivity = getActivity();
WebView web;
ArrayList<View> views = solo.getCurrentViews();
for(View view : views) {
if(view instanceof WebView) {
web = (WebView) view;
break;
}
}
WebDriver driver = new AndroidWebDriver(mActivity);
driver.get(web.getOriginalUrl());
WebElement element = driver.findElement(By.name("q"));
element.sendKeys("Test");
element = driver.findElement(By.xpath("//span[text()='Google Search']"));
element.click();

Testing strategy for mobile apps

The biggest challenge most of the mobile app development companies face is maintaining huge inventory of mobile devices which requires good amount of investment in procurement of the devices and then maintenance cost and operational cost like data usage etc. Also as new mobile devices are coming in the market every now and then (especially in case of android) so it’s practically not feasible to test app on each and every devices. Because of all these constraints mobile app testers should take balanced approach in testing.

Testing on emulators: Initial functional testing of the AUT should be done on emulators. Today most of emulators and simulators are capable of simulating the functionality of real device so first approach testers should follow is test AUT on emulator with target OS version to uncover most of the functional issues. AUT should be tested on each supported OS version.

Incase of android you can fully configure you emulator like SD card support, keyboard support etc.

Testing on real device: Once you feel that build is stable enough and most of the application specific functional and UI issues have been uncovered and fixed then you should start testing of AUT on real devices to uncover device specific issues. If you have all the devices available then you should perform testing on each device otherwise you should select required devices intelligently like one device of each target screen resolution and one device of each manufacturer, one device of each OS version. You should talk to developers to know if developers are using any OS version specific APIs etc.

Note: One important point to note here is you should select devices based on targeted mobile operator like AUT might work differently on two iPhones, one on Verizon and other on AT&T. This happens because of different technologies on both. And also sometimes because of customization done for mobile operator like one operator might block some APIs etc.

Testing on virtual devices: Today we have virtual environments like DeviceAnyWhere, PerfectoMobile etc. They have very good device inventory which you can use to test your AUT. They are very helpful as you need not to maintain huge device inventory in your organization. But as they charge on hourly basis, you need to make sure that you use them intelligently. When you have tested on app on different combination of screen resolution, os versions etc. then you could DAW etc. to sanity test your app on other devices with same configuration just to verify AUT works perfectly on them also. Before using them, make sure you have very good network connectivity otherwise you might observe some issues which might not occur on real devices.