Testing Challenges for Cyber-Physical Systems
report
Context and Motivation Cyber-Physical Systems (CPS), such as autonomous vehicles, semiconductor equipment, industrial printers, medical devices, and smart infrastructure, increasingly rely on complex software for their operation, control, and communication. Consequently, system quality, reliability, safety, and dependability are ever more determined by software quality. However, software testing, as a means to assess quality, is struggling to keep pace with the rapid growth in size, complexity, connectivity, and evolution of software. More and more, testing is becoming the bottleneck to innovation. This report analyses the trends and challenges that threaten the testability of future generations of CPS software and it identifies research directions to mitigate these threats.
Testing Challenges
• Multi-disciplinarity. CPS integrate software with physical components, which requires combining expertises, methodologies, and models from different disciplines for testing. Use of simulations and digital twins is necessary but introduces its own validation challenges.
• Size and complexity. Software size, number of components, configuration parameters, and interdependencies grow faster than testing can handle, leading to combinatorial explosion.
• Variability. CPS are delivered in many variants and configurations. While product-line engineering mitigates development effort, testing must still be performed on concrete instances, leading to another combinatorial explosion in testing.
• Evolution. Frequent changes in software demand fast and selective re-testing, but methods for test impact analysis and regression test selection are still immature.
• Component-based development and reuse. Systems combine in-house software with legacy, outsourced, open-source, and third-party components. Determining what, how much, and at which level to re-test components and integrations is a major challenge.
• Connectivity. CPS are connected and increasingly operate together, e.g., as a systemof-systems. Limited central control, independent management, missing global specifications, dynamic composition, and runtime changes necessitate testing beyond development time.
• Beyond functionality. Non-functional quality characteristics such as safety, security, reliability, usability, and availability are often non-compositional and can only be tested at higher system levels, where testing is expensive and more difficult to automate. New quality characteristics such as sustainability and fairness (of AI systems) emerge for which new testing methodologies have to be developed.
• Environment. The importance of the CPS environment increases, e.g., for safety and security testing. Testing exhaustively for all possible environments is infeasible, so selection, modelling, and simulation of realistic environmental behaviour is necessary, making environment validity an issue.
• Climbing the test pyramid. Testing CPS shifts from unit testing to higher integration and system levels, which is harder to automate and more expensive. Moreover, it requiresabstraction that introduces uncertainty and nondeterminism. Test effort shall be balanced across the test pyramid to optimize cost, coverage, and confidence.
• Agile and test automation. Large automated test suites are hard to maintain, suffer from redundancy, and may give a false feeling of confidence (“pesticide paradox”). Higher-level testing often lacks stable specifications in agile contexts.
• Runtime testing. For autonomous, self-adaptive, and continuously evolving systems, complete pre-release testing is impossible. This creates a need for after-release, runtime testing and quality control, beyond passive monitoring, and the challenge is how to do that.
• AI. Artificial intelligence is a vibrant and dynamic area with many new developments and breakthroughs happening in quick succession. This leads to many questions, challenges, and unclarities for testing and AI: how can AI be used for testing, how shall systems developed using AI be tested, and how shall systems that contain AI be tested.
• Microservices. Testing an individual microservice can be done with traditional testing methods. While testing a collection of them, many of the above trends and challenges come together, in particular, shifting testing efforts towards higher levels in the pyramid.
Research Directions
• Testability-by-design: Make testability an explicit architectural quality attribute, with testers involved early in system design.
• Model-centric testing: Develop an integrated approach to modelling, including modelbased system development (MBSE), model-based testing (MBT), digital twins, and environmental modelling, and use models as the basis for testing.
• Compositional testing: Systematically decompose testing across components, features, and versions, in an attempt to mitigate combinatorial explosion.
• Runtime testing: Extend testing into operation through active, non-intrusive runtime testing and quality control.
• AI and testing: Investigate methods for testing systems containing AI as well as for using AI to improve test efficiency and effectiveness.
• Microservices as testbeds: Use microservice architectures as structured case studies for advancing scalable testing methods.
Concluding Testing remains essential for CPS quality, however, current testing approaches do not scale to future generations of CPS, thus likely hampering innovation. Research and development for new testing methods are needed.
Testing Challenges
• Multi-disciplinarity. CPS integrate software with physical components, which requires combining expertises, methodologies, and models from different disciplines for testing. Use of simulations and digital twins is necessary but introduces its own validation challenges.
• Size and complexity. Software size, number of components, configuration parameters, and interdependencies grow faster than testing can handle, leading to combinatorial explosion.
• Variability. CPS are delivered in many variants and configurations. While product-line engineering mitigates development effort, testing must still be performed on concrete instances, leading to another combinatorial explosion in testing.
• Evolution. Frequent changes in software demand fast and selective re-testing, but methods for test impact analysis and regression test selection are still immature.
• Component-based development and reuse. Systems combine in-house software with legacy, outsourced, open-source, and third-party components. Determining what, how much, and at which level to re-test components and integrations is a major challenge.
• Connectivity. CPS are connected and increasingly operate together, e.g., as a systemof-systems. Limited central control, independent management, missing global specifications, dynamic composition, and runtime changes necessitate testing beyond development time.
• Beyond functionality. Non-functional quality characteristics such as safety, security, reliability, usability, and availability are often non-compositional and can only be tested at higher system levels, where testing is expensive and more difficult to automate. New quality characteristics such as sustainability and fairness (of AI systems) emerge for which new testing methodologies have to be developed.
• Environment. The importance of the CPS environment increases, e.g., for safety and security testing. Testing exhaustively for all possible environments is infeasible, so selection, modelling, and simulation of realistic environmental behaviour is necessary, making environment validity an issue.
• Climbing the test pyramid. Testing CPS shifts from unit testing to higher integration and system levels, which is harder to automate and more expensive. Moreover, it requiresabstraction that introduces uncertainty and nondeterminism. Test effort shall be balanced across the test pyramid to optimize cost, coverage, and confidence.
• Agile and test automation. Large automated test suites are hard to maintain, suffer from redundancy, and may give a false feeling of confidence (“pesticide paradox”). Higher-level testing often lacks stable specifications in agile contexts.
• Runtime testing. For autonomous, self-adaptive, and continuously evolving systems, complete pre-release testing is impossible. This creates a need for after-release, runtime testing and quality control, beyond passive monitoring, and the challenge is how to do that.
• AI. Artificial intelligence is a vibrant and dynamic area with many new developments and breakthroughs happening in quick succession. This leads to many questions, challenges, and unclarities for testing and AI: how can AI be used for testing, how shall systems developed using AI be tested, and how shall systems that contain AI be tested.
• Microservices. Testing an individual microservice can be done with traditional testing methods. While testing a collection of them, many of the above trends and challenges come together, in particular, shifting testing efforts towards higher levels in the pyramid.
Research Directions
• Testability-by-design: Make testability an explicit architectural quality attribute, with testers involved early in system design.
• Model-centric testing: Develop an integrated approach to modelling, including modelbased system development (MBSE), model-based testing (MBT), digital twins, and environmental modelling, and use models as the basis for testing.
• Compositional testing: Systematically decompose testing across components, features, and versions, in an attempt to mitigate combinatorial explosion.
• Runtime testing: Extend testing into operation through active, non-intrusive runtime testing and quality control.
• AI and testing: Investigate methods for testing systems containing AI as well as for using AI to improve test efficiency and effectiveness.
• Microservices as testbeds: Use microservice architectures as structured case studies for advancing scalable testing methods.
Concluding Testing remains essential for CPS quality, however, current testing approaches do not scale to future generations of CPS, thus likely hampering innovation. Research and development for new testing methods are needed.
TNO Identifier
1017232
Publisher
TNO
Collation
38 p.