Design of a forecast tool based on keystroke and mouse dynamics to understand workplace emotions and mental workload

Abstract

Individual emotions can significantly impact personal well-being and team emotional environments can affect overall team performance. In general, a team with high Emotion Intelligence (EI) usually exhibits higher teamwork effectiveness and performance. In addition, one’s emotions in a team can affect others’ emotions. Thus, in order to have a positive team emotional state (environment) and avoid one’s negative emotions propagating to others, a team emotion forecast tool is needed for teamwork management and individual emotional well-being.

This research investigated how to design an easy-to-use team emotion forecast tool. It first explored the relationship between user features and keystroke, mouse and touchscreen (KMT) dynamics features, which supported the design of a personalised emotion recognition framework. Secondly, taking physical keyboard and mouse as an example, this research designed an emotion recognition tool based on keystroke and mouse dynamics (KMD). The design followed the user-centred design process, including iteratively understanding context of use, user needs and concerns, concept design, prototyping, and evaluation. Thirdly, this study proposed to recognise mental workload and emotional states jointly as both factors influence KMD. Consequently, it investigated the interplay among KMD, mental workload, and emotional states in a controlled lab setting, leading to the development of a sequential recognition model, which was then integrated into the tool. Fourthly, a field study was conducted in a real-world office environment to collect extensive KMD data, alongside self-reported mental workload and emotional states. Various machine learning and deep learning models were trained with this dataset for comparing their effectiveness and selecting the most suitable model for further feasibility evaluation. Finally, a feasibility study assessed the tool’s usability, usefulness, and ease of use.

This study contributes to the knowledge in the following aspects: 1) Understanding of the relationship between a keyboard user and key KMT features, leading to a personalised emotion recognition framework. 2) Identification of users’ needs and concerns regarding a smart emotion recognition tool based on KMD for understanding workplace emotion and mental workload. 3) A novel model for sequential recognition of mental workload and emotional states. 4) A novel smart tool for recognising mental workload and emotion in team environments. 5) A benchmark KMD dataset for mental workload and emotion recognition.

Date of Award	27 Feb 2025
Original language	English
Awarding Institution	Northumbria University
Supervisor	Sheng-feng Qin (Supervisor) & Wai Lok Woo (Supervisor)