Multimodal grounding in HRI using two types of nods in Japanese and Finnish

Abstract

To achieve efficient and unambiguous interaction, it is essential that users and conversational robots establish common ground appropriately during dialogue. This requires the robot to compare the information received from the user with its own knowledge base and to provide feedback that reflects this comparison. In this study, we examined how two types of nods used by a robot as feedback—up-nods and down-nods—affect users’ perceptions of the robot’s knowledge state in two culturally distinct contexts: Japanese and Finnish. In the experiment, users conveyed information to the robot, after which the robot provided multimodal feedback consisting of the two types of nods and verbal tokens (or users watched a video of such an interaction). Based on this feedback, users judged whether the robot had prior knowledge of the information. As a result, in both languages, users were able to infer the robot’s knowledge state from the type of feedback provided. In Japanese, down-nods consistently conveyed the impression that the robot already possessed the information. Up-nods alone did not show a clear tendency, but a significant difference was observed between up-nods and down-nods. Although the results for Finnish did not reach statistical significance, the data suggested a similar trend to Japanese, indicating the possibility of culturally shared interpretations of nodding behavior.