Using MediaPipe to track upper-limb reaching movements after stroke: a proof-of-principle study

Emerging work supports the use of artificial intelligence-based markerless motion capture systems to complement standardized clinical measures when assessing post-stroke motor recovery. MediaPipe Pose Landmarker is an open-sourced, machine learning tool, requiring only one camera, which can be used to track upper limb movements and quantify kinematics. Here we aimed to test the use-case of MediaPipe Pose Landmarker in tracking upper limb movements after stroke in a 2-dimensional cartesian coordinate space. Participants (N = 7, > 2 months after stroke, upper extremity portion of the Fugl-Meyer Assessment (FMA-UE) range of 40–66) engaged in five sessions of a previously established, semi-immersive, gamified reaching task, involving movements of the hand and arm. Movements at four time points (first and last block of session 1 and 5) were captured by a video camera, with videos processed through the MediaPipe Pose Landmarker pipeline to extract coordinates of effectors of interest and subsequently analyze kinematic outcomes (related to movements of the hand, shoulder, and trunk). Kinematics of the hand (mean palm speed, palm bivariate variable error; BVE, where a low BVE reflects greater consistency), shoulder (BVE), and trunk (BVE) were extracted for each individual, separately across time points. Exploratory analyses indicate increased mean palm speed and palm BVE across time points. Further, analyses suggest that shoulder and trunk movements may contribute to improvements in hand-related outcomes for some individuals. Overall, our findings provide support for the use of MediaPipe Pose Landmarker in tracking upper limb movements in individuals with motor impairment after stroke.