Towards Advanced Emotion Modelling: Mixed Emotion Generation and Recognition Using Advanced Deep Neural Network Architectures

Abstract

This thesis explores the generation and recognition of mixed emotions through advanced deep learning frameworks, contributing to the field of affective computing by addressing the complexity of human emotions. Unlike traditional systems that classify emotions into discrete categories, this research focuses on the nuanced and multifaceted nature of mixed emotions—an area crucial for developing empathetic AI systems that can better understand and respond to human experiences. The motivation stems from the need for AI systems to move beyond simplistic emotion detection and account for the overlapping and composite emotional states often experienced in real-world scenarios, such as feeling both joy and anxiety simultaneously.
The thesis is structured across key chapters, each contributing novel methodologies and insights. Chapter 3 introduces a synthetic data generation framework, leveraging advanced generative models like DCGANs and image fusion techniques to address the scarcity of diverse datasets in emotion recognition. Chapter 4 presents pattern recognition techniques using a novel Local Binary Pattern (LBP) variant, demonstrating superior accuracy and robustness for emotion detection in challenging conditions. Chapter 5 introduces AffectiveFusionNet, a multimodal emotion recognition model that integrates facial expressions, speech, and text using a combination of Visual Transformers (ViTs) and Variational Autoencoders (VAEs), achieving a 91% accuracy rate on benchmark datasets and addressing limitations in unimodal approaches. It further explores pose-invariant facial expression recognition through the MOEO algorithm and LBP, enhancing robustness against occlusions and pose variations.
The core technical contribution is the development of MEG(FE)1, a first-generation mixed emotion synthetic dataset, and a corresponding detection framework that achieves 95% accuracy in recognizing composite emotions. This system was rigorously validated through convergence analysis, ablation studies, and runtime evaluations, showcasing its significance for real-time applications in human-computer interaction, virtual assistants, and mental health monitoring. Abstract By advancing synthetic data generation, multimodal integration, and deep learning-based recognition techniques, this thesis bridges significant gaps in affective computing, offering a comprehensive framework for understanding and modeling mixed emotions. These contributions pave the way for future research into empathetic and adaptive AI systems capable of addressing complex human emotional states.

Date of Award	30 Apr 2025
Original language	English
Awarding Institution	Northumbria University
Supervisor	Kamlesh Mistry (Supervisor) & Naveed Anwar (Supervisor)