By Lisa Guerriero
Software vendors rely on efficient testing solutions to ensure faster time to market and customer satisfaction. To meet the demand for shorter development lifecycles, testing products and teams must keep pace with how quickly software and hardware are evolving.
Overall, the software testing industry is broad and encompasses many back- and front-end processes. Focus areas include, but are not limited to, compliance, code, functional, integration, interoperability, load, mobile, mainframe, performance, regression, security, stress, usability, and unit testing. Many of these processes are automated to provide faster and more efficient results.
“Today’s enterprise is engaging customers on many fronts and in ever faster cycles. That is creating applications with multiple client interfaces, built on top of years of legacy systems and data that are connected with a wide variety of integration technologies,” explains Dennis Moya, product manager, automated software quality, IBM.
Although automation is essential in many cases, it is also expensive. Therefore, developers should be strategic in determining which areas are best suited for automation. Nikhil Kaul, product marketing manager, SmartBear Software, suggests that organizations place focus on automating unit, regression, integration, and performance testing in order to keep costs down. “Exploratory or manual testing is typically used to test areas either too expensive or too hard to automate,” says Kaul.
Many vendors offer software and services to support developers in these testing processes. Look to our Target Chart on page 26 for a comprehensive view of solutions and focus areas, while we highlight specific offerings and differentiating factors, as well as touch on trends in this article.
Automated Offerings
Test automation and management software and services improve quality assurance (QA) and functionality throughout the application development lifecycle. The push for multi-channel environments emphasizes the need for efficiency and quality. Testing automation solutions and services are developed to improve a variety of testing processes, offering varying levels of expertise.
CIGNEX Datamatics provides managed services for various testing types, covering significant functional and non-functional testing scenarios. Specific to automated solutions, the company uses various tools and frameworks for both Web and mobile automation to help its clients and QA teams accelerate test automation to generate a higher ROI.
Ronak Shah, practice head, QA/software testing, CIGNEX Datamatics, suggests that a key differentiator for its service is that it is entirely designed around open-source tools and products. “These solutions are incorporated well within the continuous integration process environment and suitable for all types of software development lifecycles, they can be customized as per-individual enterprise requirements.
CollabNet offers functional, regression, unit, and Web testing. The solution is a reference implantation that also uses open-source products and integrates it with application lifecycle management (ALM) product, CollabNet TeamForge, for test management.
Janardhanam Venkat, director of engineering, CollabNet Inc., notes that the product is unique because it can extend the TeamForge capability to integrate with any test management tool using the open architecture that the CollabNet enterprise agile adoption roadmap is built on.
Coverity features code, load, performance, usability, and QA testing solutions. The company offers an on-premises solution in addition to its cloud-based Scan and Code Spotter offerings. Dennis Chu, senior product manager, Coverity, says its solution offers accuracy of analysis, “leading low false-positive rates, proven enterprise scalability, as well as ROI, ease of use, and deployment.”
Hewlett-Packard (HP) provides a number of test automated solutions that cover functional, performance, security, service test automation, and network virtualization. Genefa Murphy, VP, product marketing, application delivery management, HP Software, says many factors differentiate HP from competitive offerings, including flexible deployment models, integration into its developer eco-system, and industry-only network virtualization capabilities.
“Within the HP Network Virtualization capability, automated and optimized results are built into the solution, providing code-level recommendations to enable applications to perform at least 40 percent faster,” says Murphy. “This can be built into customers’ continuous integration and deployment solutions, so automated testing and optimization recommendations are delivered as an automated result.”
IBM features a range of test automation capabilities and services. Moya states that its primary solution is the Rational Test Workbench family of offerings, which includes three components—a test authoring platform, performance test server, and service virtualization offering.
“There are two things that distinguish IBM’s capabilities, the breadth of our solutions and the flexibility we provide in consuming our offerings and services,” says Moya.
On the technology front, the company plans to continue to integrate test automation capabilities into its Rational Test Workbench to support an increasingly multi-channel world. “We can use the same user interface (UI) test across mobile and desktop, combine different types of tests into a multi-channel test scenario, and automatically generate an HTTP protocol level test from a Web UI test to drive large scale performance tests,” he adds. “This level of reuse and integration is critical to improving productivity and accelerating testing when test and delivery cycles are becoming increasingly shorter.”
InfoStrech focuses on mobile, regression, Web, interrupt, certification, firmware, and language testing. Manish Mathuria, CTO, InfoStretch, says the company can offer a selenium automation framework as well as QAs—also known as QMetry automation studio. “Both solutions provide automation capabilities at various stages in the development process from inception to monitoring and substance.”
The company brings together a blend of solutions and services to provide a complete offering, specifically focusing on the role of its solution in the mobile space. “Unlike the world of software for fixed devices, mobile application development—especially for enterprises—requires both a service element as well as a product,” says Mathuria. “Our model allows users to keep up with the dynamic nature of mobile testing. Products alone will not suffice. When it comes to mobile, it’s a whole solution value chain consisting of people, products, and processes.”
Borland, a Micro Focus company, offers the Borland Silk family, which is a suite of integrated test solutions that cover functional/unit, performance, mobile, and cross-browser testing, it can be orchestrated and managed from a single-management framework.
Tom Fisher, senior manager, product marketing, Borland, a Micro Focus Company, comments that it is the extension of integration and automation that differentiates its product from competitive point-product solutions. “The Borland Silk family capabilities and integration provide a complete range of options to satisfy any test requirement—both technical as well as financial.”
Microsoft’s Developer division offers its Visual Studio solution with a Team Foundation server, providing test planning, test case management, and manual and automated testing. It sends high-fidelity defect reports to cloud-based or on-premises ALM solutions including Virtual Studio Online or Team Foundation Server.
“Visual studio online is free to get started and Microsoft just released Visual Studio Community 2013, a free, fully supported version of Visual Studio for non-enterprise users, making the industry’s most extensive integrated development environment more easily accessible to every developer,” says Brian Harry, corporate VP, Microsoft Developer Division, Microsoft. He explains that Visual Studio, Team Foundation Server, and Visual Studio Online have products appropriate for all developers, including both startups and enterprises, and allow teams to build applications based on specific software needs.
Neotys offers performance and load testing tools. The company has two software solutions for Web and mobile applications, NeoLoad for load testing and NeoSense for performance monitoring. Both NeoLoad and NeoSense enable development, quality assurance, and IT operations to quickly and efficiently test and monitor the quality, reliability, and performance of applications. The company also provides service packages for organizations who want testing and monitoring experts to give them the results they need quickly. “More than 1,500 organizations globally have selected our solutions because they are agile, automated, easy to use, and support all rich Internet applications and mobile technologies,” comments Jonathan Bracken, VP marketing, Neotys.
He explains that if mobile users are accessing your applications, NeoLoad provides test realism with built-in device simulation and network emulation. “With its continuous integration server integrations and ability to integrate with Selenium, Perfecto Mobile, and others for end-user experience metrics under load, NeoLoad enables testers to automate a holistic view of the application’s performance. “With the recent launch of NeoSense performance monitoring, Neotys now offers the ability for QA, operations, business, and IT teams to see a single version of application performance through the sharing of testing and monitoring profiles and insights. The integrated Neotys Cloud platform enables testers to create a new level of geo realism in their testing while providing them with the power to select from a wide variety of cloud service providers to provision their load generation servers within minutes,” says Bracken.
SmartBear’s automated testing solution, TestComplete, allows developers to create, maintain, and execute automated tests for desktop, mobile, and Web. Kaul says users don’t have to be an experienced testers to create robust automated tests.
Additionally, the company pursues an unbundled strategy for Test Complete, meaning that desktop, Web, and mobile licenses are each sold separately. “Competitors sell these three licenses as one bundled solution, which makes breaking even on a solution cost lengthier,” says Kaul.
If a tester wants to run multiple tests on different machines, SmartBear’s TextExecute offering is available, which Kaul describes as a less expensive mobile/Web/desktop license that allows testers to achieve distributed testing at a reduced cost.
SOASTA’s testing solutions—including CloudTest, TouchTest, and mPusle—are targeted at eliminating delays associated with a traditional testing approach. “Focused on speed and results, SOASTA empowers development, test, and operation teams to rapidly validate mobile functionality, performance to any scale, and real user behavior in a continuous way with contextual information targeted at specific user needs,” says Brad Johnson, VP, product marketing, SOASTA.
One area that separates SOASTA from the competition is its realistic test scenarios. “SOASTA has a large, geographically distributed test platform that can be used to inject load from various locations around the world,” says Johnson. “More importantly, our platform can be used to recreate load scenarios faced by enterprises, which include ramping and spikes.”
The company also leverages its real-time online analytical processing engine to provide actionable intelligence that allows testers to drill into live data while tests are running to pinpoint performance bottlenecks in a Web application.
Telerik, a Progress Company, features the TelerikTest Studio for functional UI testing for Web and desktop applications, providing support across popular browsers and operating systems. The company also provides Telerik Mobile Testing, which is a cloud-based solution that allows tests to be created and executed against a variety of popular mobile devices.
Daniel Levy, director of product strategy, ALM, Telerik, suggests that Test Studio offers users a “refreshing approach for teams to collaborate around tests.” He explains that some edge-case tests may require developers’ assistance to script, and the Test Studio approach allows for small pieces of the test to be scripted, while keeping the remainder of the test in a format that traditional QAs can easily maintain outside of code. “Tests are passed between testers and developers via source control,” he adds.
Tricentis’ Tosca Test suite offers business risk optimization, model-based test automation, test data management, and service virtualization. Wolfgang Platz, CPO/founder, Tricentis, says the model-based solution extends beyond script-based methods of record and play to provide testing automation in a more robust, cost effective, and scalable way.
“The Tosca Test suite also assists enterprises in prioritizing and creating the right test cases that contribute to their business risk and guarantees higher risk coverage within a shorter execution time,” says Platz. “Once achieved, Tosca helps manage test data efficiently, further helping to cut down on time and effort and ensuring effective test automation.”
Worksoft provides business validation software for packaged enterprise applications. Its flagship product, Worksoft Certify, automates business process validation across all enterprise applications and interfaces—including Web-based technologies, hybrid cloud environments, big data, and mobile platforms.
“Worksoft is the industry’s only platform for automating end-to-end business process validation across the entire enterprise,” reports Shoeb Javed, CTO, Worksoft. “We offer true end-to-end business process validation for all major technologies and platforms, both SAP and non-SAP systems.”
Supporting Automation for the IoT
A significant trend in application development is the need to be cross platform. Therefore, testing is often critical across desktop, Web, and mobile devices. Automation across unit, regression, integration, and performance testing helps improve processes as well as reduce development time.
Greater automation capabilities change the entire testing climate. Enterprises require automation for functional and unit testing such as regression, as well as for performance, load, integration, mobile, and cross-browser testing. “Automation for these test categories are popular, as all enterprise applications require functional and performance testing and most are now offering access for Web, mobile access, or both. They enable testers and developers to keep up with the fast pace of releases, especially in agile development environments,” observes Borland’s Fisher.
Current testing tools provide a range of benefits. “Testing solutions save time in regression testing, give better ROI, help in early identification of defects, provide greater accuracy in catching defects, aid in application health checkup, and deliver quality solutions consistently,” summarizes CIGNEX’s Shah.
Efficiency is another priority of test automation. If less time is spent on regression testing, there are shorter release cycles, and the number of bugs that slip into production is minimized, says Telerik’s Levy. “A lower percentage of development resources is now dedicated to finding/fixing bugs.”
Because the scope of testing is so wide, enterprise processes must be checked across Web, mobile, and custom applications, and interact with numerous other platforms. “Validation is more than systems testing. Think of validation across multiple enterprise applications that encompass your end-to-end business process, not just process segments in a single application,” points out Worksoft’s Javed.
Tricentis’ Platz says automation can save money, but warns that it is different for every industry and that enterprises need to make sure the tools that they select best suit their needs. “Without a testing methodology and proper test data management, enterprises are faced with the challenge of having to sometimes create 2,000-plus user test cases that have no contribution towards their business risk,” he cautions.
Mobile Requirements
As previously noted, cross-platform needs are increasingly in demand. Mobile and wearable technologies are the newest to the scene. Many describe this area as their fastest-growing sector.
Neotys’ Bracken points out that in many cases, testing organizations are forced to cobble together several tools to achieve a truly automated solution for multi-channel environments. “Improvements have been made to integrate device simulation, real device testing, network emulation, and cloud-based testing, but there are still few vendors who make this easy and accessible,” he notes.
“Due to the increased importance of mobility and overall user experience, we see a large amount of focus placed on mobile testing both for simulation and real device testing,” observes HP’s Murphy.
Because of this, vendors that normally offer traditional solutions have expended into cloud-based offerings to support mobile testing. Agile development methods are favorable for this environment.
With the expansion of the Internet of Things, specialized testing strategy is in demand. “There is a remarkable need for fast, efficient, and agile quality assurance processes in the machine-to-machine space,” notes InfoStretch’s Mathuria.
The agile testing methodology is favored because it promotes shorter lifecycles. “Instead of waiting until the application is ready, continuous integration systems play a key role in enabling faster delivery. Continuous delivery turns integration testing on its head. Rather than integrating code at the end of a project or even a sprint, continuous integration makes code integration uninterrupted, occurring consistently throughout the project,” explains SOASTA’s Johnson. “Minimizing the number of changes integrated at one time makes it easier to isolate and correct the source of any integration bugs.”
Continuous integration is a key component of the approach because bugs are detected and resolved earlier in the process—often more affordably. “In addition, testing code quality, identifying security vulnerability, finding check style issues, identifiying broken links, and testing services in case of distributed applications can be scheduled to run through the continuous integration server,” notes Venkat.
However, utilizing agile methodology requires that both the tester and client be on board, and that the approach fit the company’s needs. Often, vendors use a combination of agile and more traditional testing methods. “Rather than avoid risk by exhaustive testing, we need to find the right balance between risk and agility, supported by continuous testing and delivery processes,” warns IBM’s Moya.
Agile methods can be a sea change for many test teams and clients. Johnson says some enterprises avoid the pain points by adopting a turnkey solution. “Combining continuous integration with continuous delivery requires addressing performance as a first-class citizen and treating it more as a feature than an afterthought,” he offers.
Complexity Challenges
As multi-channel environments become the norm, complex coding requires new attention to testing.
Coverity’s Chu says that while enterprises are more aware of the need for testing solutions because code is increasingly complex, programming tools are also more complicated. “As software toolchains become more varied and complex, integration of test tools into the various key aspects of the software development lifecycle become increasingly more important to ensure adoption success—such as source change management, bug tracking systems, desktop integrated development environments, and build tools,” he points out.
Cloud adoption also complicates automation beyond initial implementation. “The cloud can signify speed, increased usage, and scalability so automation needs to keep up with the rate of cloud adoption,” explains Mathuria.
Careful strategy and execution are critical to successful automation. “Testing solutions provide the ability to automate repetitive tasks, thereby potentially reducing human error, leading to better quality software. That said, there are limitations to automation testing. Proficiency is generally required to write automation test scripts, and test maintenance can sometimes become a costly endeavor,” notes Microsoft’s Harry.
Technical knowledge is not enough for testers. Much relies on a tester’s ability to embrace new approaches, maintain automation suites in the face of application updates, and overcome emerging code integration challenges. “Automation is expensive and requires a significant amount of dependency on automation engineers. Some may also question the reliability of automation itself, so proven solutions are important to vet out,” says Mathuria.
Many vendors emphasize the need for a mindset change among testers as well as developers and other stakeholders. “It’s increasingly hard to reproduce the real world in the test environment. Service virtualization is a major weapon against this complexity, but testers also need to collaborate with developers so they can operate at the speed of production. This implies a move to smaller and faster cycles of incremental releases, and a shift in focus from finding defects prior to shipping to verifying and fixing defects reported by early users,” says IBM’s Moya.
Harry agrees, stating that adjusting to a non-waterfall development model is often an implementation challenge. Integration points aren’t usually sequential, which enables quicker response to problems, but also means stakeholders play a different role in the process. “One of the biggest challenges of software automation today is in changing this deeply entrenched model of software development in businesses and getting the groups to be more collaborative,” he argues. “Those working with the tooling need to fully understand the capabilities of the tools, stay transparent and open with all stakeholders involved in the development process, and work to keep test scripts updated as needed.”
The Best Test for the Job
A testing solution—whether it’s a product, service, or combination—must ensure reliability despite the preponderance of platforms and interfaces. With an abundance of testing choices on the market, the most successful vendors embrace new methods and technologies leveraged to the enterprise’s benefit. SW
