TY - JOUR
T1 - Multimodal integration for data-driven classification of mental fatigue during construction equipment operations
T2 - Incorporating electroencephalography, electrodermal activity, and video signals
AU - Mehmood, Imran
AU - Li, Heng
AU - Umer, Waleed
AU - Arsalan, Aamir
AU - Anwer, Shahnawaz
AU - Mirza, Mohammed Aquil
AU - Ma, Jie
AU - Antwi-Afari, Maxwell Fordjour
N1 - Funding information: This study acknowledges the following funding grants: 1. General Research Fund (GRF) Grant (15201621) titled “Monitoring and managing fatigue of construction plant and equipment operators exposed to prolonged sitting”; 2. General Research Fund (GRF) Grant (15210720) titled “The development and validation of a noninvasive tool to monitor mental and physical stress in construction workers”; and 3. Research Institute for Intelligent Wearable System (RI-IWEAR) - Strategic Supporting Scheme (CD47) titled “An automated assessment of construction equipment operators' mental fatigue based on facial expressions”.
PY - 2023/10/1
Y1 - 2023/10/1
N2 - Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites.
AB - Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites.
KW - Construction equipment operators
KW - Construction safety
KW - Machine learning
KW - Mental fatigue
KW - Multimodal data
UR - http://www.scopus.com/inward/record.url?scp=85165367459&partnerID=8YFLogxK
U2 - 10.1016/j.dibe.2023.100198
DO - 10.1016/j.dibe.2023.100198
M3 - Article
SN - 2666-1659
VL - 15
JO - Developments in the Built Environment
JF - Developments in the Built Environment
M1 - 100198
ER -