Architecture and technical approach for DT-supported AI-based training and system optimization

WP3 is concerned with the development of a technical approach and a reference architecture for DT-supported AI-based system optimisation. System optimisation can be performed by connecting AI to both the physical system and its DT. By allowing the AI to take control over the DT, a learning cycle based on reward and punishment can be constructed to validate its actions. In order to establish a baseline for improvements, the state-of-the-art for existing solutions in AI-based system optimisation is reviewed. A framework for a cost-benefit analysis to compare methods will be constructed. The outcome of this task, combined with the state-of-the-art and prior requirements, is to produce a technical approach and a reference architecture for supporting DT-supported AI-based system optimisation suitable for use in industry use cases. This report describes the state-of-the-art in reinforcement learning and digital twin-based learning, their application in the ASIMOV use cases, as well as a reference architecture to construct and build DT-supported AI. A low-threshold introduction to reinforcement learning and Q-learning is followed by an extensive and well-structured literature overview. The initial ideas about which techniques and approaches to use and how to apply them in the use cases are then described. Afterwards, the report deals with practical challenges by concentrating on the application to specific use cases. It puts the findings in the perspective of the developed reference architecture, and it summarizes the challenges and way forward towards the end goals: apply it to the real system, make it extendable and scalable, provide the details of the technical approach, reference architecture and tools and technology that may support the building of a practical application. Chapter 4 concentrates on two use cases for UUV. The first use case addresses creating optimal test plans for testing vehicles on a test bed: improve data quality by adapting the test plan in a way that every tested scenario contains as much valuable new information as possible. The second use case addresses sensor optimization: how to tune sensor and perception parameters in a way that the vehicle can perceive its environment in the most accurate way. Chapter 5 concentrates on the TEM use case. The use case addresses part of the parameter settings of the microscope that influence the quality of the final image by reducing the aberrations caused by electron beam deviations: astigmatism, spherical aberration, coma, etc. The research starts from the rich literature on DRL and the findings in D3.2 and moves to development of a prototype AI agent that may be linked to an actual TEM system. Several techniques have been investigated, implemented and tested. The results form the basis for (1) testing the solution in the real world of an electron microscope, (b) a step towards further improvement of the case and extension to other aspects of the microscope and (c) a more formal description of the technological approach and reference architecture. Chapter 6 introduces an overarching view of AI implementations, referred to as the AI architecture. The chapter includes a reference architecture for AI implementations, containing generic elements, aspects and best practices. The content is aligned with the DT architecture discussed in WP2 and the full system architecture discussed in WP4. This document ends by drawing overall conclusions on DT-supported AI-based training and system optimization.