A Multimodal Emotion Recognition Enhanced Resilient Cognitive Twin

In next-generation engineered systems, particularly those involving complex interactions among cyber, physical, and socio-technical components, human intelligence (HI), as a "human-in-the-loop" element, needs to play a critical role in ensuring the system resilience and right-decision making under unforeseen and uncertain evolving conditions. Although predefined artificial intelligence (AI) models can offer some decision support for predefined problems, but due to its limited explainability, HI is still key to decision adaption/adoption. However, HI’s cognition and judgment are inevitably influenced by a subject’s emotional states, especially when unforeseen emergency situations happen in complex cyber-physical-social environments. To address this challenge, this study presents a critical review of Human-Cyber-Physical System (HCPS)’s resilience challenges and accordingly proposes a Multimodal Emotion Recognition enhanced Resilient Cognitive Twin (MER-RCT) of a HCPS and its three major mechanisms in addressing the system resilience. This MER-RCT not only supports the generation, interaction, and integration of human and artificial cognition and decisions under situation awareness but also incorporates emotion-aware capabilities to reflect the evolution of human affective states under situational awareness. By embedding near-real-time MER into the situational awareness and decision-support processes, the proposed MER-RCT enhances the emotional and cognitive resilience of HCPS during emergencies. A prototype MER-RCT system is validated through a flood emergency scenario, demonstrating its feasibility in distributed, uncertain environments.