Sequential gesture mapping into expanded meaning for robots

Abstract

Robotics is continuously expanding and ways of interacting and controlling robots are always being explored. Gestures have emerged as one such method where the bodily motions of a person are captured using a sensor and then mapped into some meaning for the robot to follow.

This thesis provides a pipeline for capturing sequences of gestures, which can be mapped into complex meaning. This is done simply by recognizing individual gestures and concatenating them into a sequence. This is built on top of open source methods. First is Google’s MediaPipe, which is used to recognize hands and gestures on a camera feed. Second is a Robot Operating System 2 integration of MediaPipe, which allows for the information to be passed into a node that handles sequence capturing. The experiments showed the method to be reliable and intuitive for users, although it was not implemented into an actual robot or human-robot setting.