NLP-Based Speech Interfaces in Medical VR : A Pilot Study on Usability and Interaction
Nayak, Mohit (2024)
2024
Master's Programme in Computing Sciences and Electrical Engineering
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
Hyväksymispäivämäärä
2024-12-12
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-2024120510827
https://urn.fi/URN:NBN:fi:tuni-2024120510827
Tiivistelmä
Applications of virtual reality (VR) have grown in significance in the medical field because of it’s ability to recreate real-life scenarios in three dimensions while posing no risks to patients. However, there are several obstacles to overcome when moving from 2D screens to 3D VR environments, including the complexity of the visual data, complexity in controls, and adaptation in 3D environments. These difficulties often lead to ineffective user experiences, so more intuitive interaction techniques are needed.
In response, our research explores the potential of intelligent speech interfaces to enhance user interaction while conducting complex medical tasks, like planning surgery for Dental Implants. We designed and developed a voice assistant within VR, leveraging intent recognition of Natural language techniques to answer questions and also interpret user intentions and execute tasks such as obtaining different surgical equipment by converting their verbal inputs into actions and getting voice feedback on completion.
To evaluate the usability and cognitive load of this speech-based interaction, we conducted a mixed-methods within-subjects user study with 20 student participants. The study compared the voice-assisted approach to traditional interaction methods such as buttons, across various tasks. Data Collection comprised of structured questionnaires and open-ended responses, which indicated that speech-based and button interfaces demonstrated similar performance in terms of usability and cognitive demand, with each modality displaying distinct strengths.
Our findings indicate that NLP-driven voice assistants can enhance interaction and accessibility in medical VR, especially in areas such as locating controls, ease of control, user comfort, and intuitive interaction. These findings highlight the potential benefits of augmenting traditional controls with speech interfaces, particularly in complex VR scenarios where conventional methods may limit usability. This study identifies key areas for future research, including improving the intelligence, accuracy, and user experience of speech-based systems. Addressing these areas could facilitate the development of more robust, user-centric, voice-assisted applications in virtual reality environments.
In response, our research explores the potential of intelligent speech interfaces to enhance user interaction while conducting complex medical tasks, like planning surgery for Dental Implants. We designed and developed a voice assistant within VR, leveraging intent recognition of Natural language techniques to answer questions and also interpret user intentions and execute tasks such as obtaining different surgical equipment by converting their verbal inputs into actions and getting voice feedback on completion.
To evaluate the usability and cognitive load of this speech-based interaction, we conducted a mixed-methods within-subjects user study with 20 student participants. The study compared the voice-assisted approach to traditional interaction methods such as buttons, across various tasks. Data Collection comprised of structured questionnaires and open-ended responses, which indicated that speech-based and button interfaces demonstrated similar performance in terms of usability and cognitive demand, with each modality displaying distinct strengths.
Our findings indicate that NLP-driven voice assistants can enhance interaction and accessibility in medical VR, especially in areas such as locating controls, ease of control, user comfort, and intuitive interaction. These findings highlight the potential benefits of augmenting traditional controls with speech interfaces, particularly in complex VR scenarios where conventional methods may limit usability. This study identifies key areas for future research, including improving the intelligence, accuracy, and user experience of speech-based systems. Addressing these areas could facilitate the development of more robust, user-centric, voice-assisted applications in virtual reality environments.