Feeling the beat: a smart hand exoskeleton for learning to play musical instruments

Individuals who have suffered neurotrauma like a stroke or brachial plexus injury
often experience reduced limb functionality. Soft robotic exoskeletons have
been successful in assisting rehabilitative treatment and improving activities of
daily life but restoring dexterity for tasks such as playing musical instruments
has proven challenging. This research presents a soft robotic hand exoskeleton
coupled with machine learning algorithms to aid in relearning how to play the
piano by ‘feeling’ the difference between correct and incorrect versions of the
same song. The exoskeleton features piezoresistive sensor arrays with 16 taxels
integrated into each fingertip. The hand exoskeleton was created as a single unit,
with polyvinyl acid (PVA) used as a stent and later dissolved to construct the
internal pressure chambers for the five individually actuated digits. Ten variations
of a song were produced, one that was correct and nine containing rhythmic
errors. To classify these song variations, Random Forest (RF), K-Nearest Neighbor
(KNN), and Artificial Neural Network (ANN) algorithms were trained with data
from the 80 taxels combined from the tactile sensors in the fingertips. Feeling the
differences between correct and incorrect versions of the song was done with
the exoskeleton independently and while the exoskeleton was worn by a person.
Results demonstrated that the ANN algorithm had the highest classification
accuracy of 97.13% ± 2.00% with the human subject and 94.60% ± 1.26% without.
These findings highlight the potential of the smart exoskeleton to aid disabled
individuals in relearning dexterous tasks like playing musical instruments.