Chapter 4 by Broadcom
Business ●●○○○
Technical ●●●○○
Listen to the audio version
AI and machine learning techniques enable dynamic optimization of test suites at scale as part of the continuous integration and delivery lifecycle.
Introduction
In an age of Agile and DevOps, agility of software delivery is of paramount importance. In section 1 chapter 1 Quality Engineering and recent trends, we previously discussed how testing as the final activity in the workflow is unquestionably the biggest bottleneck in delivering high-quality code. Sub-optimal testing not only incurs more effort and costs, but also results in lower productivity, release delays with potential negative business impact.
Currently, model-based testing techniques are used to perform static optimization of tests via requirement modeling. However, static optimization ignores dynamic conditions such as changes to specific bodies of code, previous test run history, test flakiness, and other runtime criteria (code churn rate, code quality, number of authors who modified the code, and more).
In fact, except for certain circumstances, tests optimized using model-based testing may be suboptimal – implying that we are not doing everything possible to reduce testing effort and cycle time. On the other hand, trying to make test optimization decisions manually based on a plethora of such data would be prohibitively time consuming and non-scalable.
This chapter discusses how such solutions can be implemented using customer case studies as examples. We demonstrate how we leverage AI/ML to generate dynamically optimized test sets through the use of the following techniques, among others:
Broadcom's Continuous Delivery Director (CDD) Test Advisor product incorporates these approaches.
A recent Google case study revealed an astounding rate of development among 25,000 software engineers:
We also see comparable data from other organizations. According to a recent case study conducted at Microsoft, the Windows development team made an average of 8500 code pushes per day, resulting in approximately 1760 builds! And Facebook's mobile application receives approximately 1500 code pushes per day.
Now consider what happens in terms of testing as a result of this type of frenetic development activity.
To put the volume of testing into context, let's return to the Google example. Their corresponding testing numbers are as follows:
Given that the majority of enterprise software engineering organizations are not as large as Google's, we normalize this data on a per-build basis for each developer. This equates to approximately two code builds per developer per day, with an average of 1,500 automated tests per build.
Given that the majority of other enterprises are unlikely to be as mature as Google, let us assume an average of one build per day, resulting in approximately 750 tests per build. Given that the majority of organizations do not use significant test automation, let us assume that 50% of the 750 tests are automated and that manual tests take an average of 5 minutes to execute. This equates to approximately 32 hours of manual testing per build, not to mention the additional effort (and time) required to prepare test data and environments for these tests.
Thus, for a typical agile team of approximately eight people (say, five developers), this equates to more than 160 hours of manual testing per build! This is a significant overhead that could result in release delays or, more likely, testers failing to test adequately, increasing the risk of failure.
This clearly demonstrates how difficult it is for testing to keep up with the development pace in an agile environment. While techniques such as model-based testing would likely help reduce this burden by about 50%, they would require additional modeling effort and may still be insufficient to keep up without fully automating all tests. As a result, we must consider novel ways to address this issue through the use of AI/ML.
There are numerous ways in which AI/ML can assist in resolving this issue, including correlating multiple data sets associated with the testing process. All of these contribute to the reduction of test volume to an essential subset based on the various approaches described below.
Let’s look at three different approaches of Broadcom’s Continuous Delivery Director (CDD) Test Advisor solution.
Associating Tests with Code changes
Given the frequency with which code is changed, a critical approach is to determine which tests are impacted when specific code is changed. By instrumenting the code under test, we can establish a link between the tests being run and the code being tested. The AI system is trained (and continues to learn as more tests are executed) by running existing tests against the code base.
Thereafter, whenever code is changed (via a commit to the source code management system), Test Advisor flags the tests that are impacted by the change, prioritizing their execution.
Naturally, all new (or updated) tests would be qualified automatically as tests that must be run as part of a build.
Understand pattern of past test results
An AI system can deduce patterns from previous results to determine which test sets should be prioritized for execution. One of these patterns is what we refer to as flay tests. These are test suites that exhibit one or more of the characteristics listed below:
If the test suite's results show a discernible trend and the last several test runs are all successful, the test is not identified as flaky. This type of test result indicates that an issue existed but was resolved.
A test can be flaky as a result of a bug in the code or as a result of a flaw in the test itself. A flaky test suite is automatically selected for the next test cycle.
Mapping defects found to code components
By associating defects discovered during tests with the application components from which they originate, an AI system can identify which builds require additional testing based on the code components included in the build. Naturally, we must prioritize defects appropriately based on their severity, location within the CI/CD lifecycle, and type. This enables the AI system to proactively identify high-risk or defect-prone components and flags out the tests associated with them (as described in feature (A) above) whenever they are included in a build.
In summary, Test Advisor assists in reducing the amount of testing required by flagging tests that must be run as part of a build. This frequently results in a reduction of more than 60% in the number of tests run (in comparison to those run using traditional approaches).
A large telecom service provider has been implementing DevOps practices to transition to an agile delivery lifecycle. Naturally, they were concerned about the volume of tests they needed to run in conjunction with frequent code changes and builds associated with their bi-weekly release cycles. They were typically required to run over 500 tests per build of an application (including unit, component, and BDD tests, API and integration tests, and system tests), the majority of which were manual. Carrying out all of these tests would significantly slow their release cycle. They were able to reduce the volume of tests by 70% to 90% depending on the application by implementing CDD and Test Advisor. By reducing the number of tests, they were able to significantly reduce elapsed time and testing-related delays.
Looking forward
We have only begun to scratch the surface of AI/ML applications in testing and quality – and a vast realm of possibilities awaits. Several key use cases that we are considering for future enhancements include defect prediction and source identification, quantification of release/deployment risk prior to going live, and automated test (and test data) generation based on production usage scenarios.
Shamim Ahmed
Shamim Ahmed is CTO for DevOps Solution Engineering at Broadcom Technologies, where he is responsible for innovating intelligent DevOps and BizOps solutions using Broadcom's industry leading technologies.
His key focus areas include: applications of AI/machine learning to DevOps, "model-based everything" in Continuous Delivery (specifically model-based requirements, testing, deployment, release, test data and services) , site reliability engineering (SRE), and modern application architectures.
Broadcom Inc. is a global technology leader that designs, develops and supplies a broad range of semiconductor and infrastructure software solutions. Broadcom’s category-leading product portfolio serves critical markets including data center, networking, enterprise software, broadband, wireless, storage and industrial. Our solutions include data center networking and storage, enterprise, mainframe and cyber security software focused on automation, monitoring and security, smartphone components, telecoms and factory automation. For more information, go to www.broadcom.com. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
