By Lana Gates
In today’s Internet age, people have myriad choices when it comes to products and services. If a user feels that a Web site performs too slowly, for example, he or she moves on to a different one that performs faster. Loyalty in this environment is hard to come by, but it is attainable. One of the best ways to gain customer loyalty is through quality services and applications.
To develop and distribute quality applications requires thorough testing to patch up any vulnerabilities and work out any kinks prior to launch—a process that can carry a considerable cost. Testing in the cloud can have a higher initial cost than that of traditional testing methods but can save companies money over time—30 to 40 percent, according to Melinda Ballou, program director of application life-cycle management and executive strategies, IDC, Framingham, MA.
IDC sees cloud testing, which can include testing cloud-based applications, subscription-based pricing for testing, and analytics to understand software performance and functionality in the cloud—as “kind of a killer app,” says Ballou. “It’s a no-brainer. Organizations need to be able to provision infrastructure to run their test environments.”
They do that by creating test environments that replicate what they expect to experience once they put their applications into production. “If you can create testing environments in the cloud that mirror your production environment, and then you can evolve them and change-manage those environments, it saves significant time and money to be able to do that,” points out Ballou.
Cloud testing differs from traditional testing methods in that it enables organizations to provision additional servers on demand that may be located anywhere geographically. Organizations can augment their data centers and reuse those environments “but not reinvent the wheel every time,” notes Ballou.
“Traditional testing is all internal and really doesn’t bring visibility back to the customer end point,” adds Bryon Colaizzi, director of performance, Paychex. The company provides payroll, human resources, and benefits services outsourcing firm based in Rochester, NY.
Reduced costs and the instantaneous extension of a company’s servers, also known as scalability, are two drivers behind testing in the cloud. Others include speed, efficiency, elasticity, and flexibility.
Examining the Market
IDC groups automated software quality (ASQ)—which comprises all kinds of testing as well as analysis to determine security vulnerabilities and authorization for the cloud—in with cloud testing and tracked revenue for this area at $261.4 million in 2011. (See Figure 1.) The research company forecasts revenue to increase to $1.1 billion in 2016.
“In the face of a volatile global economy, reinvestment but constrained IT staffing, and complex sourcing, demand is emerging strongly for SaaS ASQ and cloud testing solutions and will continue to do so throughout the forecast period, driven by customer demand for innovative cloud delivery models for quality and offerings from both small third-party providers and larger vendors,” says Ballou.
A leader has emerged in this growing field, according to IDC. SOASTA offers performance testing through CloudTest On-Demand, load testing through CloudTest, and mobile testing through TouchTest. The company also offers SOASTA Platform, which includes CloudTest, a visual dashboard, TouchTest, real-time analytics, and Global Test Cloud.
Paychex, with 564,000 clients, enlisted SOASTA Platform when it found it was running out of data center space and needed a cost-effective financial model that would allow it to resource on demand, says Colaizzi. He calls it his “Nirvana vision.”
Using SOASTA Platform, Paychex is able to provision environments on the fly, capture results, push the results to a dashboard, and then deprovision environments, explains Colaizzi. “It’s an available, secure, on-demand resource.”
Yet security is often one of the major challenges in adopting this type of environment, especially for a financial services company. Chandranshu Singh, an analyst with Ovum, points out that “actual data is not used for testing.” Instead, he says, “testers use data masking and data generation techniques for creating dummy data to use in tests.”
It took Paychex about six months to arrive at a secure state with cloud testing. “We had to go through a lot of negotiation, requirements definition, create a proof of concept, ran a series of payroll processing in SOASTA’s environment, partnered with an organization at the time to run ethical hacking, looking for vulnerability,” explains Colaizzi.
At each step of the way, Paychex asked if its applications were highly available, transparent, cost-effective, and secure—until it felt confident they were. “The combination of all that caused the due diligence and rigor to give us confidence to proceed in that fashion,” he adds.
Paychex achieves a secure environment by keeping its applications in a private cloud within its four walls. “We don’t have our customers coming into a public cloud entry point,” says Colaizzi. The payroll company partnered with a third-party telecommunications company to conduct payroll services and processing. “We’ve done all of our standard security audits and opened up a secure channel in the cloud from our data center to their resource in the cloud as if it was inside our four walls.”
Jockeying for Position
Other leaders in the field, according to IDC, include Parasoft, with its SOAtest and Load Test offerings, which are packaged together; and Compuware, with its Gomez 360 Web Load Testing tool for load and regression testing and analysis.
Jack Henry & Associates, a technology provider for financial services, based in Monett, MO, used Parasoft SOAtest to improve its jXchange SOA-based banking product. When Jack Henry & Associates’ quality assurance (QA) department receives new code from developers, it creates test scenarios and uses SOAtest to run those scenarios. QA is able to capture and keep those results and tests for future testing.
“Being able to keep regression tests in our SQL database, and then being able to validate our responses and then store them for future regression testing is a huge benefit,” says Mary Hulett, QA manager of Jack Henry & Associates jXchange.
“If I look at the amount of time that it takes us to validate those responses the first time through, and then multiply that by how often we run the regression tests, I would say that using this feature has saved us a couple hundred hours worth of work over the past year,” adds Hulett.
Other major players in the space include Keynote Systems, a company that offers Cloud Application Perspective for performance and Web load testing; IBM, with its Performance Testing and Application Virtualization products; Hewlett-Packard (HP), with HP Performance Center; Skytap performance and load testing in Skytap Cloud; Micro Focus, with Borland Silk Portfolio, which includes Silk Performer CloudBurst; SmartBear, through SoapUI Pro, which offers functional, regression, and compliance tests; and CAST Software, with its Application Intelligence Platform. Additionally, IDC considers Microsoft a contender with Windows Azure.
CareerBuilder.com, which counts more than 14 million unique visitors on its site each month, had the need to test from multiple geographical locations over the Internet. The recruiting Web site sought help from Keynote Transaction Perspective and Keynote LoadPro in the process.
“We wanted to test our site the same way the real end users would,” explains Mark Fouraker, VP of technical operations, CareerBuilder.com. “Plus we knew that by using Keynote’s service, we could avoid making a large capital investment and taking on the job of running remote testing locations,” he adds.
Keynote testing servers housed at various locations across the globe and on different Internet service providers all loaded the site at the same time to perform a load test, beginning with 75,000 session starts an hour. Four tests and many days later, the CareerBuilder.com site successfully handled 300,000 session starts an hour with more than 45,000 concurrent users.
“Production experience has now confirmed what we saw in the tests,” points out Fouraker. “Working with Keynote helped us to move forward with confidence in our efforts to support the launches of new partners and, at the same time, save valuable dollars.”
Going Mobile
But are those results customary? Chandranshu Singh, an analyst with Ovum, believes that although cloud-based software testing is gaining in popularity, it’s not necessarily an enterprise necessity right now for all organizations and applications.
“Organizations need not follow a ‘rip-and-replace’ approach to cloud-based testing, which means move their testing effort to the cloud and spend both money and time doing that when these resources can be used for some other pressing need,” he cautions.
Some applications lend themselves better to cloud testing than others do. Those best suited for this method are Web and mobile applications, as well as compute-intensive, customer-facing applications.
Mobile technology is adding to the attraction of testing in the cloud. Singh calls mobile the “sweet spot” and says that all mobile apps should be tested in the cloud.
Because these are newer, there are no legacy issues that need to be considered. “It works out cheaper, more efficient, and also more effective than traditional testing approaches,” he says.
IDC’s Ballou agrees and highlights the need for testing of mobile applications. “Testing in mobile environments has lagged behind the helter-skelter rush to put out anything on smartphones/tablets,” she explains. “Cloud testing enables faster, more efficient testing of mobile apps.”
In fact, the rise of mobile technology is helping to drive testing in the cloud. “They’re symbiotic with one another,” notes Ballou. She attributes that to the highly demanding performance requirements of mobile applications.
Not One-Size-Fits-All
“The primary benefit of cloud testing, regardless of application type,” Singh points out, “is the on-the-fly provisioning of test environments, which cuts a lot of test execution time down as test environment setup and tear-down time is drastically reduced.” Stress and performance testing, he adds, are well suited for the cloud.
Yet other types of testing are done in the cloud as well, including compliance, load, scalability, cross-browser, and interoperability, among others.
A low level of interoperability between different clouds, however, has been a pain point. Some cloud providers, including Rackspace, IBM, Citrix, Nebula, Cloudscaling, and HP, have embraced the OpenStack standard to some degree, says Singh.
OpenStack is an open source cloud operating system that handles computing, networking, and storage in its data center, which can be managed by a dashboard. “The idea is that you should be able to take your application from one cloud to the other with minimal changes/transforms,” adds Singh. “It should simply work.”
When considering testing in the cloud, Singh cautions that organizations should be mindful of interoperability issues while choosing a cloud provider. If this is not considered the company may find themselves suffering a lock-in by the cloud vendor.
Rather than trying to test everything in the cloud, he suggests companies ask themselves, “Under what circumstances would cloud-based testing benefit my development effort?” It’s a more practical tactic.
“By answering this question,” he explains, “businesses will have a better sense of the effectiveness of cloud-based testing. Where cloud testing works, it is cheap, efficient, and the best possible way of getting the job done; where it doesn’t, it’s just a fancy way of doing testing,” he explains.
Also it is important to keep in mind that the initial setup and structure takes a bit longer than it does with traditional testing methods. In addition, “you have to measure how much you need to be testing and the types of spikes you have in demand,” notes IDC’s Ballou.
Retailers that experience increased demand around the holidays are good examples of companies that can benefit from testing in the cloud.
If outsourcing your testing, she advises retaining at least part of the quality portion of your applications/services in-house so you can track how providers are meeting your requirements.
Looking ahead, Ballou hopes to see a better way of leveraging back-end systems, in addition to a rise in standards. The landscape is still young, and there’s plenty of room for improvement. IDC anticipates continued acquisitions in this space as it all shakes out.
Despite this method’s youth, Colaizzi of Paychex terms using the cloud for testing “exact path testing.” He’s very pleased with the ability to manage the financial component at a very appreciative cost point, the maintainability, the reliability, the on-demand provisioning, the ability to drum up additional compute cycles as necessary and then tear them down, the flexibility, transparency, and agility.
“We’re now in position to create test activity that’s not geographically bound to our location,” he notes, “and allows us to understand user experiences from multiple geographies across the globe.” SW
Jul2013, Software Magazine
