By Lisa Guerriero
Mobile devices are increasingly entrenched in business, utilized in day-to-day activity and actually shaping workflows. The more enterprises rely on their devices, the more emphasis should be placed on mobile testing.
The multiplicity of mobile devices, operating systems (OSs), and networks complicate the process of testing applications (apps). Testing must account for device and platform conditions, including memory, keypad type, battery, and new data models like voice control.
Mobile testing tools and service providers tackle the challenges in numerous ways. Functional and performance testing are among the mainstays, but testing types are expanding—as well as the tools and solutions to support them.
App testing providers don’t always agree on the best approach. Emulators and simulators enable a level of repetition that many view as critical to thoroughly evaluating an app’s mobile performance against a variety of unknowns. Others emphasize real device testing, saying that no simulation can replicate the data that comes from user experiences. In-the-wild testing is growing, introducing a crowd sourcing approach to real device testing.
Often, test providers recommend testing with both real devices and simulators or emulators. Real devices are ideal for initial tests, as well as to indicate real-world look and feel, and to understand how the software functions. Simulators and emulators are most appropriate for repeatable testing.
Additionally, many vendors emphasize the importance of testing as an integrated part of an app’s lifecycle development. This is achievable through an end-to-end solution, but developers can also accomplish it with a stand-alone solution if they build the app with testing in mind.
Software Magazine examines how emulators/simulators and real devices both address mobile testing challenges, and the benefits to solutions that integrate testing into the development lifecycle. We also spotlight select tools on the market.
Testing Tools
Mobile testing tools and services providers often handle native, mobile Web, and hybrid apps. Solutions vary widely, but they all strive to help developers quickly and cost effectively bring apps to market without sacrificing quality. The prevalence of mobile devices, both within and outside the business world, signifies great opportunity for developers—but it also complicates the testing phase of the development lifecycle.
“The main issue that faces mobile app developers is the growing array of application programming interfaces (APIs) that either the device or the app server is connecting to for the ever-growing demand for mobile content. Consider four years ago that a mobile app had three system dependencies. Today, the average number of system dependencies has risen to 15,” observes Wayne Ariola, chief strategy officer, Parasoft.
Emulation and simulation are popular methods of mobile testing, especially for performance, when high-volume testing can be cost prohibitive. Because of their speed, they provide an efficient way to test small changes to an app, and are well suited to continuous integration (CI) workflows.
Ophir Prusak, chief evangelist, BlazeMeter, recommends real devices for functional and user experience testing. “On the other hand, running performance tests with real devices is impractical and in this case emulators and simulators are the better solution, particularly if you are talking about testing mobile apps with 100,000 or millions of concurrent users,” says Prusak.
Shravanthi Alimilli, product marketing manager, Telerik, a Progress Company, recommends emulation or simulation when testers are unsure which mobile devices to invest in for testing, due to budget constraints.
Additionally, emulation/simulation are a good option when there’s a short deadline to producetext execution results and purchasing the required mobile devices isn’t feasible. “It might be necessary to use the emulator/simulator in these circumstances for testing the relevant mobile apps,” explains Alimilli.
The argument against emulation often focuses on the difficulty of reproducing real-world conditions. An app’s success varies due to factors like memory, battery, keyboard type, the number of apps a device contains and is currently running, and whether the user upgraded to the most current OS.
Eran Kinsbruner, technical evangelist, Perfecto Mobile, says it’s tempting to use emulation, but warns of the risks. For example, he says it’s difficult to replicate accurate screen sizes with emulators, or to vigorously test capabilities like Bluetooth and GPS.
“When testing an app, it’s important for developers to employ the device that their user will be experiencing their product on—a device that is true to the end-user experience,” suggests Kinsbruner.
Georg Thurner, product manager, and Gerd Weishaar, VP of product management, Tricentis, caution that apps present problems that are difficult to simulate. “In most Android cases, some issues are linked to special user interface (UI) versions from vendors and providers,” they explain. “iOS can only be simulated on Apple hardware, which is even more expensive than the actual device.”
Many vendors say emulation and simulation are necessary for high-volume testing, but should be used in conjunction with real-device testing, rather than replacing it.
“Real devices always have the benefit of the ability to reproduce precise device conditions, and are sometimes the only approach to test end-to-end scenarios that need the real network, back-end services, third-party apps, and the device itself on which services are launched to test functionality, interoperability, and performance,” says Venugopal Rk, managing director, Accenture digital–mobility.
In-the-wild testing goes a step further, evaluating the app outside of a controlled lab setting. This method accumulates user data from a variety of locations and devices in varying conditions.
“Only by putting apps through paces on real devices under real-world conditions will help you understand how your app will perform in the hands of users,” says Damian Roskill, VP of marketing, Applause. He adds that the best development teams utilize a portfolio approach, with real devices as well as emulation or simulation.
Automated scripts are an important component of many testing solutions. Automated testing is now part of most solutions, or offered to complement manual testing.
“Automation is critical for repetitive results in functionality and performance. Emulation allows this in a CI environment more easily than a real device,” notes Saria Mapanao, marketing manager, Appvance.
Testing within
the Lifecycle
Providers of end-to-end solutions emphasize the need for testing to be an organic part of the larger development lifecycle. Many factors drive developers to these solutions—including shorter production cycles, the benefits of agile development, and rapidly changing mobile platforms.
It is important to have a well thought-out strategy for product development, leaving ample time for testing. With a plan in place, better testing is more apt to happen without taking a project schedule off track.
“Developing mobile apps today means having to navigate a fragmented device landscape, manage continuous update cycles, cater to always-on customers, and contend with fierce competition,” observes Karl W. Stewart, director of product marketing, SOASTA. “To keep up with these demands a continuous approach to mobile app testing is no longer an option, it’s a requirement,” he says.
Neil Patterson, technical evangelist, IBM DevOps for mobile, notes that mobile apps are usually multi-tier. The majority of enterprise apps integrate with cloud- and mainframe-based services to deliver data, security, and transactions—necessitating end-to-end, not just on the device.
In addition, end users expect mobile apps that are personalized and in context based on factors like location, preferences, and past experience. “This aspect of mobile app design must be factored in to ensure not only acceptance of the app but success over competitive apps,” explains Patterson.
Automation is a critical enabler to testing across the complete app lifecycle, according to end-to-end solution providers. It ensures the app’s quality and performance on any device, OS, network, and location, notes Antoine Aymer, worldwide mobility solutions manager, Hewlett-Packard (HP).
“Another key enabler of this strategy is device-agnostic scripting, which allows enterprises to build test cases that can be easily maintained, reused, and ported to multiple platforms,” adds Aymer.
Many vendors recommend that developers consider testing early, regardless of whether they select a stand-alone testing solution or one in which testing is part of a larger lifecycle management service.
Manish Mathuria, CTO/founder, InfoStretch, suggests a “blueprinting phase” in which the developer’s company identifies the target audience, platforms, tech locations and domains, and interface requirements with third parties and communication protocols.
“Building apps with testability in mind always helps. For example, an app that scans bar codes can provide a programmatic or an API interface to automatically provide it with images to be scanned, such that a tester does not have to manually scan hundreds of images to test,” notes Mathuria.
Jonathan Lipps, director of ecosystem and integrations, Sauce Labs, says mobile apps require testing throughout the lifecycle—in some ways, even more than desktop apps. With mobile, apps need extra time to receive approvals from app stores before they can be posted.
“A bug in the code can cause major issues and delays, even when remedied quickly. It is important that mobile apps are extensively tested before release so users get the best possible experience,” he explains.
Testing Tools and Solutions
Many software vendors provide solution that aid in mobile app testing. Here, we highlight just a few tools designed for mobile app development.
Accenture features the Testing Center of Excellence, a global organization with more than 27,000 members, including over a thousand devoted to digital testing. The organization’s testing types include functionality, performance, interoperability, conformance, device compatibility, security, language, and accessibility and user experience.
Accenture built its mobile app testing lab with its proprietary Remote, Offshore, Simulation, and Automation—ROSA—framework. It leverages components like network simulation environments, hosted device clouds with remote access, and industry-aligned reusable test components that are designed to fast track mobility deployments and defect-free rollouts. The lab supports all device platforms and app types, including native, mobile, and hybrid apps.
Applause offers a crowd sourced testing approach for native, mobile, and hybrid apps. Its services cover functional, usability, localization, load, security, and accessibility testing. The company leverages a global community of more than 175,000 professional testers in 200 countries and territories.
Fragmentation of the mobile industry combined with the number of devices in use present serious challenges to developers, says Roskill. To overcome these challenges, testing in the wild provides real-world insights.
Appvance’s automated platform covers Android and Apple’s iOS. The company offers compatibility, functional, and performance testing for mobile—including transactional performance. Its beginning-to-end solution imports and runs existing scripts, addresses scalability issues, and enables rapid analysis.
Its PerformanceCloud v2.0 solution enables quick identification of bottlenecks, driving higher performance and scalability from apps. Testing drives the app from beginning to end, at a protocol level, or both. Appvance designed it to help teams—development, quality assurance, and DevOps—leverage cloud computing, agile methodologies, and CI. It allows them to efficiently run realistic functional, performance, and load tests and triage application issues.
BlazeMeter’s automated performance testing platform addresses native, hybrid, and mobile apps. Its solution works with open source technology and runs natively on all platforms. The company offers built-in mobile network emulation capabilities. With robust API, it integrates with many development tools and processes, such as CI.
BlazeMeter recently introduced Mobile Recorder, which allows users to create performance tests by recording native apps directly from their mobile devices, even with secure connections. The new feature creates a test script by recording user activity on the device, reducing testing time from hours to minutes. It doesn’t require modifying the app, and the tester doesn’t need to jailbreak the device or install anything on it.
HP’s solutions cover a gamut of testing tools for mobile, native, and hybrid apps, including function, performance, and context. The company equips its solutions with big data analytics by utilizing its AppPulse Mobile product. It provides insights from real end-user interactions with mobile apps, allowing developers to address problems quickly and ensure a better user experience.
HP offers an end-to-end solution as part of its Application Lifecycle Management (ALM) suite, integrating mobile testing as part of the larger infrastructure. This approach ensures mobile testing meets the same quality standards and business objectives as desktop and Web testing. It enables management and tracking of all app projects, providing consistent workflows and processes. The ALM suite integrates with HP’s app security solutions, ensuring vulnerabilities are addressed during development and also once an app is in production.
IBM has multiple solutions for testing native, mobile Web, and hybrid apps. It automates functionality, UI, compatibility, and integration testing through its MobileFirst Platform, with the MobileFirst Foundation Studio product, as well as a software testers’ solution, Rational Test Workbench. IBM designed its automated mobile test scripts to ensure they are effective even for users without programming skills. MobileFirst also offers manual interactive testing—including functionality, UI, consumability, performance, and engagement. IBM recommends this product for engaging users throughout the app development lifecycle. Focused on quality assurance, it allows testers and end users to submit feedback, including bug reports, without leaving the mobile app. It also provides analytic capabilities by mining app store ratings and reviews.
The Rational Test Workbench provides service virtualization capabilities for functionality, performance, integration, and resource testing. This product emulates complex middleware test environments so that test teams can focus on the client tier of the mobile app. The workbench product also organizes and manages app testing tools for mobile front ends, supporting middleware and back-end services. It consolidates and connects test results from multiple tools into a single quality metric, and links data from test case failures back to development for defect removal.
InfoStretch offers more than 600 certified testers for enterprise-grade projects, and its test labs have over 4,000 real devices. Addressing native, mobile, and hybrid apps, its testing types include functional, non-functional, interrupt, memory leak, app performance, usability, installation, vulnerability, language testing, security, and memory.
InfoStretch also provides test automation, enabling ongoing regression with minimal effort for its customers’ teams. Utilizing technology from partners like Perfecto Mobile, it reduces the ongoing cost of regression testing across multiple devices and improves the performance of apps. Its automated testing scripts are reusable across device types, saving customers time and helping improve app quality.
Parasoft specializes in app service virtualization, though it provides a variety of other testing types for native, mobile, and hybrid apps. These offerings include API functional, API security, API load/performance, manual functional, automated functional, and load/performance. It recommends using real-world devices when possible, and utilizing simulation when it’s not feasible to test on the actual production system.
Its service virtualization solution, Parasoft Virtualize, simulates back-end transactions for a full array of devices and app servers. It supplies mobile developers with a realistic development environment, regardless of conditions like location and time.
Perfecto Mobile’s specialty is the Continuous Quality Lab, which offers an on demand infrastructure for testing mobile apps. The lab integrates with popular development tools, languages, and frameworks. It enables testing throughout the lifecycle, to accelerate release frequency and ensure product stability and quality. Designed for enterprise-grade service, the lab meets corporate security policies and offers around-the-clock support. Perfecto Mobile built it to scale as projects, requirements, or needs grow.
The Continuous Quality Lab enables automated testing of real devices located in the cloud, focusing on functional and performance testing and monitoring. It supports nearly 99 percent of the device/OS combinations on the market, whether the app is mobile, native, or hybrid. Perfecto Mobile’s ScriptOnce automation allows one script to be executed across multiple platforms and devices simultaneously. The lab is a hybrid model, allowing customers to use the cloud type that best fits their testing, compliance, and coverage requirements.
Sauce Labs provides an extensive automation cloud for testing Web, native, and hybrid mobile apps. It automates JavaScript unit and functional testing for over 500 browser and OS combinations, helping developers bring apps to market faster. Its solution is optimized for CI and continuous delivery environments.
Users can run tests written in any language or framework using Selenium or Appium, which are both widely adopted, open source standards for automating browser and mobile app functionality. Sauce Labs notes that Appium was dubbed “Selenium for Mobile,” and works with a user’s pre-existing test practices, frameworks, and tools. In addition, Sauce Labs’ solution includes videos, screenshots, and HTML logs to pinpoint problems quickly, and its Sauce Connect tool allows users to securely test behind their firewall.
SOASTA offers an end-to-end solution, utilizing several products to automate testing on real devices. TouchTest is a key tool in the solution, combining gesture-based functional testing automation with mobile device cloud technology. It creates tests quickly by capturing the user experience on real devices—playing back each gesture and action with the precise velocity, movement, and orientation as when recorded. It handles an array of complex gestures, including shake, orientation, rotate, and pinch.
TouchTest is usable on its own or as a component of a CI process. It integrates with multiple CI tools to enable lights-out automation with the check-in of every build. SOASTA offers both public and private clouds, allowing manual and automated testing across 300 real devices and comparison of multiple devices at once. The devices are protected in the cloud and automatically wiped after every use, keeping data secure.
Telerik, a Progress company, provides a functional testing framework for native, hybrid, and mobile Web apps, as part of an end-to-end development platform. It enables automated testing of iOS, Android, and Windows phone apps against any mobile browser. Customers can connect devices for execution by USB, WiFi, or by Telerik’s device cloud for access to 350 real iOS and Android devices.
Automation engineers and developers leverage Telerik’s framework to write tests in JavaScript on Windows or OSX—or via Telerik’s cloud-based integrated development environment, AppBuilder. Tight integration with AppBuilder enables seamless test execution in the device cloud. Customers write tests in plain text that can be mapped to platform-specific coded commands. Telerik’s beta device cloud enables app testing on 300-plus devices against a variety of hardware platforms, OS versions, manufacturers, and screen resolutions.
Tricentis’ Tosca Testsuite platform allows combined end-to-end tests over various technologies. The Tosca Mobile+ tool provides functional testing of mobile apps on real devices as well as emulators. It supports native, hybrid, and mobile Web apps on iOS and Android devices.
Under Testsuite’s model-based test automation approach, all supported technologies are represented and used in the same way. Users don’t have to learn new tools, and Tosca Mobile+ doesn’t require any additional ramp-up time. It enables unique IDs to create robust test automation and ensure ROI, rather than object index or image-based recognition, which the company considers less reliable.
Developing with Testing
App testing must keep pace with the radically changing role of mobile devices. The diversity of testing tools, however, rivals the scope of mobile platforms. There may not be a one-size-fits-all solution, but developers have options to suit their projects, budgets, and development methods.
A combination of testing with real devices and emulators or simulators will benefit most app developers, especially when testing is integrated into the app development process. SW
Jul2015, Software Magazine
