Challenges in Automated Mobile Testing
The mobile phone has transitioned from its traditional role as a mere medium of communication to a multipurpose personal device. Mobile testing has become a requirement and it also brings a lot of challenges.
Mobile Testing – Necessity
Quality testing of applications across all operating systems, device platforms and networks is a laborious task and the non-functional testing with usability, security and adaptability becomes a critical testing need as much as functional testing. Effective and appropriate testing helps the device manufacturers and app developers to collect metrics which in turn help in improving the overall product quality. Most applications designed for mobile platforms require the ability to run on a number of network modes as well.
One of the main factors that will determine the success for any automation is its ability to work across different platforms and on various technology stacks.
Automated Mobile Testing – Challenges
- Device Diversity
a) Different platforms with multiple browser supportb) Differences in renderingc) Varied application performance at run time for mobile devices
- Network Challenges
a) Different networks types (GSM/GPRS/Wi-Fi/Wi-Max)b) Connectivity performance across different locationsc) Different network operators with custom networks
- Hardware Challenges
a) Processing speedb) Memory size of mobile devicec) Different device communication protocols like WAP or HTTP
- Multitude of Mobile Devices
Mobile devices like smart phones, tablets, iPads and wearable devices provide a huge variety of challenges which any mobile app faces and it is very difficult for the testing team to guarantee that the application will work well on any given device.
- Device Fragmentation and Various OS Platforms
Mobile platforms are constantly evolving technology platform and the most difficult aspect of the mobile testing is the variety of devices available especially when we look at the Android platform.The difference in device and display variety is so huge that most of the testing frameworks are cross-platform to help execute test programs easily on variety of devices. These open source cross-platform frameworks such as Calabash, Appium etc. are pretty flexible and very effective solutions for multi-platform testing needs.Another major challenge in mobile applications testing is the compatibility issue when they need to be deployed across devices which have different operating systems like iOS, Android, Blackberry, Windows, etc. and each one of these operating systems having different version of the OS running on different devices.
- Mobile Application Types
With a mobile app that can be either native, web or hybrid, the testing required for each of these types is different from the other due to their implementation differences.
- Test Interfaces and Test Execution
Test execution across different devices and browsers is a huge effort as it is difficult to deploy and run. For example, consider a test case that needs to be executed on platforms like IOS, Android and Blackberry. The test case in this scenario will be expected to be executed on a minimum of 15+ different devices and in more than 10 different combinations which could lead to approximately 150 rounds of testing. It is tedious and time-consuming to execute such combinations and scale of testing.Testing tools like mobile emulators & simulators play an important role as they help verify the general functionalities and perform the regression testing but the testing conducted on emulators and simulators is not conducted in a real environment. The advantages of these tools are limited in scope and cannot be considered a substitute for real testing. Using the emulators and simulators along with in normal testing can give best results.
- Network Connection Types
Mobile data comes in with variety of connections types like edge, umts, 3G, 4G and Wi-Fi (b, g, n). Sometimes there may be no connection at all on the device. When a user is on the move, the connection type would change. Though the connection APIs on mobile platforms are developed keeping these scenarios in mind the real environment still can be different and there may be other issues that can occur. Hence testing different network switching scenarios and the bandwidth usage becomes another important factor.
On mobile platforms touchscreen interactions are quite open-ended because of variety of screen sizes and resolutions. The test for such cases can be automated but it is quite likely to find problematic cases or unexpected states while testing these scenarios. An automated testing strategy can help verify that the text in an app is displayed as expected but it can’t determine whether that text is semantically clear and user-friendly. The usability testing on the other hand can only be accomplished with manual testing and with the tester’s strategy to capture the minute details visually.