Non-verbal communication in instant messaging: conveying emotion through voice interfaces

Abstract

Instant Messaging has become a keystone of human personal communication, where the biggest application WhatsApp is currently serving over 2 billion people. Plenty of research confirms people use non-verbal communication in computer mediated communication, allowing for emotional communication at distance. At the same time, Virtual Personal Assistants, such as the Google Assistant and Apple Siri, are continuously expanding their market share. Recently, they have included support for voice-based instant messaging, which includes reading aloud instant messages. As instant messages are synthesised, included digital non-verbal communication traits may be lost or omitted. This study aims to explore the impact of text-to-speech conversion of instant messages by virtual personal assistants on recognition of non-verbal cues by the receiving party. Secondly, the research aims to explore and test methods to include non-verbal communication traits in instant messages to speech synthesis, by the inclusion of spatial arrays (emojis) and modification of synthetic voice prosody. Sentiment analysis and emotion detection are explored and applied to extract emotional data from instant messages, which can be used to modify speech synthesis characteristics, such as pitch and speech rate, to mimic human paralanguage and vocal non-verbal communication to convey emotion.