By Olivia Cahoon
Application (app) development processes require testing to reduce risks and software vulnerabilities. Continuous testing employs automated testing as part of the software delivery to provide instant feedback to developers.
Continuous testing strategies meet business requirements and addresses risk coverage so the time to market is reduced, allowing developers to release updates earlier. Here, we discuss the evolution of this testing method as well as the end goal of implementing continuous testing.
Drivers for Testing
Continuous testing tools are developed to speed up the testing process and produce an efficient pipeline. The demand for continuous testing methods in application development stems from businesses that pressure development teams for faster delivery—accelerating the adoption of agile DevOps practices. In addition, quality assurance (QA) is at risk for falling behind, requiring a change in methods for integrating testing into the delivery pipeline.
Kelly Emo, director, app development, test and delivery, product and solution marketing, MicroFocus, believes increasing software complexity drives the need to test earlier in the lifecycle process as apps evolve to become more compositions of services—requiring the need for more API testing earlier. “There are more configurations and surface areas to test on as apps deploy across a wide array of browsers, platforms, and devices,” she explains.
To mitigate the risk of software failures, developers opt to test software as often as possible throughout development and delivery. James Hunter, program director, DevOps industry solutions, IBM Hybrid Cloud, says that test automation is a fundamental part of DevOps that ensures regular feedback of the state of software through continuous testing. This ensures teams test as early as possible, continue to test for the project’s duration, and minimize the impact of testing through the use of automation.
While organizations are under pressure to deliver software faster than ever, continuous integration (CI) and continuous delivery (CD) tools have risen to popularity. Greg Lord, director of product marketing, SmartBear Software, says that CI/CD can’t thrive without continuous testing, otherwise the QA process becomes a bottleneck in the overall development and delivery process—forcing organizations to make tradeoffs between speed and quality.
Furthermore, increased pressure on user experience and immediate feedback on app experience drives the need to test across quality spectrums early in the lifecycle. According to Emo, manual processes are not able to keep up.
Chris Kraus, product manager, Worksoft, believes that changes for production systems comes in two variations—code changes and configuration changes. Code changes are designed for custom app development and system integration, which changes when packaged and hosted apps have updates to consume. “Configuration and code changes occurr more often than ever because the consumerization of IT has caused business users to expect faster delivery,” says Kraus. This results in users demanding the same level of new features and workflows in their work life as they have in their digital home life.
Evolving Over Time
Nearly five years ago, continuous testing began to evolve. Test executions were once automated until developers realized they needed to automate more than execution for continuous automated testing.
Anthony Edwards, CTO, TestPlant, says achieving continuous testing required test data generation, service visualization, and test environment management to dynamically allocate SUTs to test and configure. “Testing tools themselves had to become more robust to irrelevant changes so minor layout and code changes don’t impact the user,” he explains.
Traditional test automation tools are designed to repeatedly confirm if outcomes at check points are matching the excepted outcome. “QA-level test automation is traditionally performed at the user interface (UI) level through scripting or record and playback tools,” explains Wayne Ariola, CMO, Tricentis.
To avoid false positives as an app evolves, most UI-level tests are constantly updated. When apps change frequently, test maintenance becomes time consuming and teams eventually abandon test automation efforts and revert to manual testing. Ariola says that as a result of these difficulties, few teams have achieved satisfactory levels of test automation—even in waterfall processes. “Industry research indicates that average test automation levels have been hovering around 18 percent for years,” he adds.
As today’s industries demand more from testing, changes across the industry make test automation more difficult to achieve with traditional methods and tools. “Test automation is designed to produce a set of pass/fail data points correlated to user stories or app requirements. Continuous testing on the other hand, focuses on business risk and providing insight on whether the software can be released,” says Ariola.
While continuous testing tools have evolved over time, some believe the biggest change is the combination and integration of key tools that enable DevOps for continuous development and testing.
Mark Creamer, CEO/president, Conformiq, believes continuous testing starts with automated test execution but without integration with development test management, requirements management, and test execution tools, it is not an option.
End Goal for App Development
The end goal of implementing continuous testing as part of an app development process encompasses several factors.
Jeff Scheaffer, GM, continuous delivery, CA Technologies, says there are three main goals of implementing continuous testing—delivering an excellent app experience to the end user, ensuring the efforts to improve quality don’t impact time to market, and achieving test coverage at speed.
MicroFocus’ Emo agrees with Scheaffer’s first goal, adding that “assuring constant, predictable, and optimal user experience with increased ability to deliver new capabilities to achieve business objectives ahead of the competition and new digital disruptors entering the market is the end goal of implementing continuous testing.”
While the first goal may be obvious, the second encompasses the entire belief behind continuous testing. “It’s being able to insert testing and test automaton practices into every phase of the SDLC to uncover and repair unexpected results as soon as they are injected into the system,” says Scheaffer.
He says the third goal focuses on achieving development velocity while reducing risk. “Have we comprehensively tested all of the software to validate that it will work and perform at the required level?” asks Scheaffer. This lets business know that systems perform at any traffic level or spike while meeting the end user’s requirements.
Kraus believes that the end goal is simple, find the software defects that break business processes and prevent business continuity before going into production. “There are always defects and bugs in custom and packaged app software, we just need to ensure they are not part of and impacting the mission-critical business workflows,” he says.
Implementation
A critical component of implementation and ongoing management of enterprise apps is ensuring end-to-end processes work. Kraus says continuous testing is an essential phase of continuous delivery process and helps address the demand for superior customer experiences. “We gain immediate and real-time feedback on business risks and maintain continuity across the business—something fundamentally necessary and the foundation for successful business outcomes,” he says.
“Continuous testing expedites software development and delivery through continuously executing automated tests as part of the software delivery pipeline to obtain immediate feedback,” says Emo. It also reduces the risk for software defects by monitoring challenges during continuous testing for immediate solutions.
According to Hunter, the key benefit of automated and continuous testing is to increase test coverage and reduce the impact of testing through automation. “Wider test coverage and more frequent testing improves a project’s chance of identifying problems and so mitigates risk to the business of software failure,” he says.
Despite the processes’ ability to meet business requirements and understand risk coverage, continuous testing is still met with challenges. One of the most obvious limitations of this testing method is the level of change required to adopt risk-driven continuous testing.
“Achieving process maturity with continuous testing requires a concerted change effort focused on people, process, and technology,” says Ariola. He believes the focus on true process transformation has been ignored in relation to software testing.
Karl Stewart, senior product manager, web experience, Akamai, says that from a product manager’s point of view, understanding the completeness of the product as a whole is one of the most notable challenges of continuous testing. “Breaking development and testing into smaller pieces makes it harder to understand the state of the whole product at any one time.”
Creamer believes functional or business risk testing is not done thoroughly in an agile development process because the test design and execution process can’t keep up. He says it’s transformational and requires change while undermining the existing team software process business model, which sells bodies. “Ultimately, the app developers will be forced to change or the competition will put them out of business because they can’t keep up,” he adds.
Continuous testing requires true automation with no gaps and yet some automated tests require a form of manual intervention at various points. Edwards says that doesn’t work. Continuous testing methods are challenged with preparing the entire testing environment including the proper SUTs, configuring SUTs, setting up test data, and selecting the right tests to run.
Another challenge is ensuring tests are robust enough to execute regardless of change while still monitoring important concerns. “That’s where testing from the user perspective helps—ignoring implementation details that the user doesn’t care about—being able to set tolerances in the same way a human would,” adds Edwards. To achieve this, he says smart algorithms help determine the success and failure of an app.
Trends for Testing
While continuous testing tools are consistently updated and improved to automate DevOps, the trends that impact these tools are also constant.
Chris Colosimo, product manager, Parasoft, believes that the entire continuous testing space is being disrupted with free. “I love free, but free has a cost in the form of long-term strategic deficiencies,” he admits. While the continuous movement changes every day, he believes there should be a stronger focus on core competency—what the testing solution is doing to support continuous integration.
Additionally, the initiative of widespread environments is gaining attention. “Why deploy into my real, complex, shared, often broken environments? If I only changed this one thing do I really need the whole environment to test my change?” asks Colosimo. No, end users only need the app and database or critical interface. Most of the other dependencies can be simulated.
Colosimo says users can create bubble environments—collections of virtual services on a temporary server with the app deployed in the middle for testing. “If all of your unit API tests pass, then you can integrate it into the larger environment and run the full stack. You will have a much higher degree of confidence that it will not explode, and if it does, you will have more information to use to make an informed analysis and solve the problem,” he explains.
To some, artificial intelligence (AI) is the next big trend that will affect continuous testing. “It is no longer a matter of debating whether AI will replace automated testing, but how soon it will happen,” says Emo. As app complexity increases, test coverage grows linearly. She believes automation testing is crippled with repetitive tasks like scripting and maintenance, which makes it difficult for test engineers to keep up with testing.
With AI, human-powered testing won’t be eliminated, rather that it will focus on identifying problems that require a creative subjective creativity. “AI will also face its own challenge, such as interpreting human behaviors based on data to be able to generate the right output,” adds Emo.
Overall, Stewart believes automation is continuous testing’s mantra as unit tests, functional tests, and regression tests are now included in every build. “There has been recent challenges in the industry—as evidenced by websites going down under heavy usage—to test the entire end-to-end performance of the product using continuous testing as currently defined,” he offers. With an array of different tests contributing to every build, testing functions are now more well-equipped to handle continuous testing than ever before.
Preventing Risk
Constant demands of apps by businesses require a fundamentally new approach to identify and prevent business risks associated with software failures. In order to meet these ever-changing and increasingly complex demands, continuous testing tools are implemented by development teams to relieve pressure from businesses for faster app delivery, to satisfy the need for superior customer experiences, and reduce failure rates.
Nov2017, Software Magazine
