Multimodal Smart Glove System Based on Flexible Optical Fibers and Inertia Sensor Hybrid

Hand motion capture devices are widely used in sign language translation, virtual reality (VR) motion reconstruction, and human–computer interaction. However, current devices often suffer from limited hand information collection due to the use of single sensor types, large sizes, and unreasonable layouts. To address these issues, we propose a smart glove based on the fusion of optical fibers and inertial sensors, which consists of nine polydimethylsiloxane (PDMS) flexible optical fiber sensors and an inertial sensor. The flexible optical fiber sensors detect finger joint bending, while the inertial sensor captures spatial hand information. The system employs a multimodal fusion algorithm to enable both gesture recognition and human–computer interaction. Dynamic gestures of the 26 English letters and common English words consisting of two to five letters can be recognized using long short-term memory (LSTM) or bidirectional long short-term memory (BiLSTM) models combined with an attention mechanism. The gesture prediction accuracy for the 26 English letters and common English words reaches 99.76% and 97.66%, respectively. Additionally, the glove system wirelessly transmits data to a computer via the ESP32S3 chip’s Wi-Fi module, where UE4 software uses C++ programming to achieve VR motion restoration. By encoding the glove data in Python and transmitting it to a robotic hand, real-time control of the robotic hand can be achieved.