Multimodal Interaction leveraging non-expert coverage extensions: DR 4.3
report
The present report describes the work carried out in the third project year regarding Natural Multimodal Interaction. It summarises the Deliverable D4.3: “Natural Multimodal Interaction leveraging non-expert coverage ex-tensions”.
In year 3, many additional functionalities have been added to the overall system, which are also supported by appropriate natural language interac-tions, which needed a significant amount of resources from this work package. A first open-source release of the dialogue management framework VOnDA has been deployed, which is complemented by a visual debugging and editing tool.
An episodic memory module was implemented that analyses the data base and refers back to past episodes (e.g., activities by the child) during the use of the MyPal app. This is a first step towards using memory-based interactions that show remembrance by the virtual agent.
As a preparation for the next phase, a prototype for off-activity task based on a symbolic memory and a prototype for robust interpretation us-ing machine learning components has been implemented. To prepare for the higher load on the database, on one hand because there are now more mod-ules actively monitoring the database, and on the other hand because of the increasing amount of users, we did research in the area of stream reasoning, that can provide efficient updates and results for our components even in a highly dynamic system with a bigger data load.
In year 3, many additional functionalities have been added to the overall system, which are also supported by appropriate natural language interac-tions, which needed a significant amount of resources from this work package. A first open-source release of the dialogue management framework VOnDA has been deployed, which is complemented by a visual debugging and editing tool.
An episodic memory module was implemented that analyses the data base and refers back to past episodes (e.g., activities by the child) during the use of the MyPal app. This is a first step towards using memory-based interactions that show remembrance by the virtual agent.
As a preparation for the next phase, a prototype for off-activity task based on a symbolic memory and a prototype for robust interpretation us-ing machine learning components has been implemented. To prepare for the higher load on the database, on one hand because there are now more mod-ules actively monitoring the database, and on the other hand because of the increasing amount of users, we did research in the area of stream reasoning, that can provide efficient updates and results for our components even in a highly dynamic system with a bigger data load.
TNO Identifier
973794
Publisher
Pal Personal Assistant for Healthy Lifestyle ; TNO
Collation
340 p.