Multimodal integration for data-driven classification of mental fatigue during construction equipment operations: Incorporating electroencephalography, electrodermal activity, and video signals

Imran Mehmood*, Heng Li, Waleed Umer, Aamir Arsalan, Shahnawaz Anwer*, Mohammed Aquil Mirza, Jie Ma, Maxwell Fordjour Antwi-Afari

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

9 Citations (Scopus)
49 Downloads (Pure)

Abstract

Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites.
Original languageEnglish
Article number100198
JournalDevelopments in the Built Environment
Volume15
Early online date13 Jul 2023
DOIs
Publication statusPublished - 1 Oct 2023

Cite this