15 Years of World Quality Report: A Journey in Quality Engineering and Testing

In the first edition, four topics were surveyed: Testing Tools and Technology, Testing Organization and Tester Profile, Outsourcing Trends, and Economic Impact of Testing. The world was just recovering from the financial crisis. There had been a focus on cost cutting. At that time, most organizations were focusing on industrialization of software development and quality assurance (QA) activities. They were doing this by creating enterprise-wide QA teams, with a focus on shift-left and automation as the main drivers that helped achieve shorter and higher quality life cycles.

In the second edition, we saw the emergence of agile and cloud. We found that 60% of the surveyed companies leveraged agile, although most were still experimenting. There was an uptake of test automation, but many companies struggled to realize ROI from test automation. And 50% had already begun with the early steps of using cloud architecture for test environments. We noticed that organizations were mostly developing their own standards for testing.

In the third year, we analyzed for the first time testing budgets. We found that a meagre 18% of IT budget was spent on testing, which was an indication of insufficient attention on quality. More organizations adopted the Testing Center of Excellence approach to bring about efficiency and scale their benefits. But it was clear that testing was mostly to be considered a cost factor and that organizations had difficulties seeing the true business value of quality.

In the 2013 edition, we noticed that quality and testing became more important to businesses. We noticed a transition from testing to a quality assurance mindset. The quality assurance budgets rose from 18% to 23%. There was further streamlining and centralizing of QA activities to achieve efficiency and cost optimization. Mobile testing emerged as a new specialization. Still, a major issue was that projects typically started planning for testing activities too late, and that project teams hardly reported on business-oriented quality metrics.

The following year, we saw the rise of transformational programs using social, mobile, analytics, cloud, and the Internet of Things (IoT). This changed the focus and increased the importance of QA and testing. The spending on QA and testing increased to 26%, as a result of the proliferation of digitalization initiatives. Most organizations were now following a hybrid approach to QA and testing, with a combination of centralized service centers and embedded test teams in projects. Organizations kept on looking for faster and more responsive QA and testing solutions that are integrated with agile development methods. Test-driven development (TDD) was on the rise. But in these agile teams, a true risk-based strategy approach for testing was lacking.

Digital Transformation Era

In 2015, we noticed a huge push for digital transformation of businesses. But organizations clearly struggled with the efficiency and effectiveness of testing in agile transformation programs. The spending on testing rose sharply to 35% of IT budget. The main reason for the inefficiency in testing was insufficient adoption of quality strategy and testing expertise that could keep up with all the digital transformation demands. In most agile teams the testing was insufficient and too slow. Agile teams expected more technical skills of test engineers and we saw the emergence of the SDET role (Software Development Engineer in Test). Test-driven development and behavior-driven development (BDD) became standard practices. Virtualization solutions helped teams become more efficient with test environments.

In 2016, we noticed a further shift in focus – it was now more toward how QA can improve business outcomes. There was a clear differentiation between the core IT and the nimbler mobile and web applications (bi-modal IT). Test environments and test data were key bottlenecks that were preventing testing from becoming more efficient. QA budget spent dropped to 31% but still was not at the level of efficiency it should be.

By 2017, agile was a standard practice and DevOps emerged as the next faster way of working. The challenges with testing in these agile teams were increasing as organizations intended to adopt the DevOps ways of working rather radically. In DevOps transformations, the role of the quality engineer was easily overlooked. The budget spent on testing dropped to 26%. Test automation appeared to become a bigger bottleneck in the transformation to agile ways of working. Many organizations were looking for smarter end-to-end automation platforms, as they assumed that that would be the solution to meet ever shorter deadlines. And we saw for the first time the introduction of smarter analytics and autonomous testing solutions. These promising solutions appeared to allow teams to keep up with an ever-faster pace of change.

With the transition to agile and DevOps, many organizations in the 2018 survey had moved away from quality guidelines and quality standards. The general belief was that autonomous teams would always find the best way forward by themselves. A key finding was that we noticed a back-to-basics approach and a focus on quality and testing. But the challenges with testing in agile teams were increasing. Lack of data and challenges with test environments were specifically called out. Test data and test environments were the clear achilles heel for many agile teams.

There was also a demand to do more with AI. The role of data scientist, and the ability to use data analytics for improved decision-making, picked up. The time was ripe to balance the freedom of autonomous teams with some more structured guidelines and tool choices, to optimize costs better.

Coping with global challenges

In 2020, the world was hit with the COVID-19 pandemic. Around the time of the release of WQR in September, most organizations and economies showed a remarkable resilience to cope with the crisis. The pandemic clearly fuelled digital transformation programs. And it forced organizations to create new ways of working with team members collaborating remotely. This proved to be very efficient, and the adoption of agile and DevOps continued to grow. We saw encouraging progress in quality engineering practices. QA teams were transforming into enablers and orchestrators of quality within agile teams. We also noted a clear uptake of test automation, which was becoming more intelligent and comprehensive. The pandemic disrupted the uptake of artificial intelligence and machine learning, causing a slowdown of true incorporation of AI in QA and testing. The uptake of cloud, virtualization solutions, test data creation, and test management tools also helped to make significant progress in test environments and test data.

In 2021, the world emerged from stringent lockdowns and saw a greater appetite for innovation. Organizations wanted test automation to be faster, with greater quality, and with more agility. Artificial intelligence was gradually being infused into the QA process. We also saw continued growth in the ability of organizations to spin up test data and test environments on demand. Security initiatives picked up. And we saw the rise of the intelligent industry: 5G, augmented reality, and virtual reality were becoming integral parts of business operations.

Last year the world left the pandemic behind, but new and potentially even greater challenges emerged: geopolitical instability, supply chains breaking down, a serious shortage of skilled resources in almost all domains, growing global inflation rates, a potential economic recession, and continued environmental and social challenges. To stay successful, organizations needed to be highly responsive to change. Also, they needed, more than ever, to focus on generating value for their customers. Continuous change was required. Value outcome was the objective. Agile development and digital transformation continued to be the key drivers for further investments in IT. And we saw the emergence of sustainable IT as dedicated focus. Only 72% of organizations thought that quality could contribute to the environmental aspect of sustainable IT. If organizations want to be environmentally sustainable, they need to learn to use available resources optimally. A stronger strategic focus on quality is the way to achieve that. Quality needs to be a core part of the strategy to drive more sustainable IT and it again is a key theme in the latest edition of the WQR. All these developments have direct implications for the continued development of both IT quality and software testing. “The closer you look, the more you see” was the theme of last year’s World Quality Report: by probing deeper and utilizing appropriate tools and resources, we can gain a better understanding of whether IT solutions will provide a benefit to our end customers and achieve valuable outcomes in terms of business performance. This, for us, is the essence of quality engineering.

And now in 2023 we have launched the 15^th edition of the World Quality Report. Clearly, the times are more challenging. Organizations focus more on cost, and investments are delayed. But when it comes to quality, organizations realize more than ever that quality of software and IT is of critical importance. It is not only about speed, but about value at speed. Quality engineering needs to be faster, smarter, and more cost-efficient. And a new major disruptor has emerged: generative AI systems. These powerful systems  bring a lot of potential benefit , to all kinds of areas, including software quality engineering  activities. However, the survey also shows that organizations need to apply generative AI step by step, with enough expertise to ensure sufficient reliability, effectiveness, and efficiency of the outcomes.

The future outlook

Over the past World Quality Reports, we have seen serious transformations in the way organizations assure quality outcomes of IT developments. From traditional development and testing methods to highly agile and continuous quality engineering approaches. From mostly manual testing to smart, automated, autonomous, AI-powered quality solutions. The importance for organizations to embed end-to-end quality engineering specialists to ensure quality is now at the core of business operations. It is fundamental to success and not a ‘nice to have’ option.

New, more powerful technologies have emerged to assist us in making decisions. But one thing remains clear: organizations always need trust and confidence in outcomes. Organizations demand clear added value to business outcomes from new technologies. And everything must be ethical, sustainable, and secure. This only proves that there is a never-ending need for fast, efficient, and effective validation of quality of software and products.

Download the 15th edition of the World Quality Report now.