Autonomous Real-Time Testing

Thomas Fehlmann, International Senior Researcher and Consultant - Euro Project Office AG

Software testing is becoming increasingly important because more and more products are software-intensive. Cars, for example, contain control software (ECUs) that is networked with each other. However, software problems can delay commissioning by months, even years, because the different components are not coordinated with each other. A timely system test would help, but there is a lack of time and resources. The functionality of the software is simply too great for manual testing. So, you must automate.

Automation is not only necessary for the execution of tests, but above all for the generation of suitable test cases. You cannot manually write test cases for a functional size around a million function points. However, this becomes possible with Combinatory Logic, the Analytic Hierarchy Process (AHP), and Quality Function Deployment (QFD).

ART – Autonomous Real-time Testing – is more than test automation. When today’s cars use map services from the cloud, or their own sensors, for an Advanced Driving Assistance System (ADAS) to perform driving decisions; or when in the future an autonomous car meets another; or with truck platooning; or when adding a new, previously unknown device to an IoT orchestra, the original base system expands its functionality. Therefore, such an expanding system needs being retested before it can do decisions with the potential of affecting harm to humans or things, after each update, after each learning. This is Continuous Testing during operation; it supplements Continuous Delivery and Continuous Integration.

Disruptive innovations in automotive require an equally disruptive new approach to testing of software-intense systems. This requires moving from once-upon-a-time testing before release to autonomous real-time software & systems testing during operations, with indications to users and suppliers about the actual state and testing results.

This presentation explains the theory and the implementation approach for a framework for Autonomous Real-time Testing (ART) of a software-intense system while in operation.

Continuous testing of complex systems in real-time requires limiting the number of test cases being generated and executed. Automatic test case generation requires avoiding combinatorial explosion. This becomes possible by understanding the needs (or values) of the customer, or user. The hype for autonomous cars is over but ADAS are state of the art and will appear in vehicles everywhere. They are likely to contain deep learning devices such as Visual Recognition Systems that share learning among a wide range of cars.

ART addresses the need to automatically generate test cases; based on an initial set of test stories. Using Six Sigma Transfer Functions, we can effectively limit the growth of test cases and keep focus on the users’ needs. Neither QFD nor Six Sigma is widely known among automotive, software testing, or DevOps engineers, that is what is exciting for attendees of this presentation.