Low-Fidelity Motion-Captured Hand Visualization for Piano Teaching through Asynchronous Augmented Reality Collaboration

Abstract

In piano teaching, hand posture and fingering techniques are hard to teach in online or pre-recorded classes. We explore asynchronous remote collaboration for piano teaching, overlaying a virtual reconstruction of the teacher's hands on top of their student's physical piano. We present a novel system that uses hand-tracking as lowfidelity motion-capture (MoCap) and MIDI data as a 'quick and dirty' method to record and playback hand movement on a piano, using only an augmented reality head-mounted display. The design and implementation of the tool are guided by expert recommendations and a preliminary accuracy evaluation. The usability is evaluated by 5 piano players and experts, in a qualitative experiment. The system achieved a mean accuracy of approximately 85%, demonstrating promising potential while still requiring further refinement. Issues in accuracy were noticed by the participants but the system is received with enthusiasm by experts and users and was still viewed as generally usable for the intended tasks.