Multimodal Human–Robot Collaboration in Industrial Cable Harness Installation Task
Jokinen, Lauri (2026)
2026
Automaatiotekniikan DI-ohjelma - Master's Programme in Automation Engineering
Tekniikan ja luonnontieteiden tiedekunta - Faculty of Engineering and Natural Sciences
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Hyväksymispäivämäärä
2026-05-25
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202605256286
https://urn.fi/URN:NBN:fi:tuni-202605256286
Tiivistelmä
Human–robot collaboration (HRC) is an emerging subfield of robotics, focused on scenarios where a human operator and a robot are working together in a shared workspace and manipulating shared objects simultaneously. It can allow for optimal deployment of both human and robotic capabilities in industrial assembly and manufacturing tasks and is therefore increasingly researched. Multimodal human–robot interaction (HRI)—such as speech recognition and generation, visual gesture detection, and force sensing during co-manipulation—can offer a human-friendly way of communication during HRC tasks, while also facilitating the robot’s ability to obtain comprehensive view of the working environment. While classic robotics has mostly been dealing with rigid objects, in recent years the interest towards deformable object manipulation (DOM) has increased, as many applications where modern robotics should be applied require handling of flexible materials.
This thesis investigates multimodal HRC in tasks involving DOM, and particularly, is focused on co-manipulation of deformable linear objects (DLOs) such as cables. The topic is studied via a practical industrial use case, dealing with collaborative installation of a heavy cable harness as part of hybrid car assembly, and a literature review, addressing the state of the art in the field of DOM and aiming to find optimal solutions for the practical use case.
In the practical part of the thesis, an initial prototype of the cable harness co-manipulation system is developed following an agile trial-and-error approach. The prototype is tested in a practical laboratory setting to observe its practical performance and identify its major defects. The literature review is then conducted—guided by the challenges faced with the initial prototype. Lastly, a future development plan towards the complete cable co-manipulation system is derived from the review findings and the experimental results describing the strengths and weaknesses of the initial prototype.
The developed prototype uses computer vision for human gesture detection and cable harness state estimation and deploys speech recognition to obtain voice commands from the user. Based on these inputs, the system controls a collaborative robot arm to manipulate the cable harness in the physical world.
In the practical experiment, the system reached an overall success rate of 0.38 with per-feature success rates ranging from 0.72 to 1.00 in an end-to-end cable-handover sequence. We consider these results sufficient for proof-of-concept purposes: while far from perfect, they prove the feasibility of our approach.
The major defects of the system include unstable grip, compromised handover accuracy, and occasional force-related hardware failures. The future development plan includes deploying force-based compliant control combined with vision-based optimization to address the hardware failures and poor handover accuracy, and integrating a customized gripper specifically designed for cable handling to improve the grip performance.
This thesis investigates multimodal HRC in tasks involving DOM, and particularly, is focused on co-manipulation of deformable linear objects (DLOs) such as cables. The topic is studied via a practical industrial use case, dealing with collaborative installation of a heavy cable harness as part of hybrid car assembly, and a literature review, addressing the state of the art in the field of DOM and aiming to find optimal solutions for the practical use case.
In the practical part of the thesis, an initial prototype of the cable harness co-manipulation system is developed following an agile trial-and-error approach. The prototype is tested in a practical laboratory setting to observe its practical performance and identify its major defects. The literature review is then conducted—guided by the challenges faced with the initial prototype. Lastly, a future development plan towards the complete cable co-manipulation system is derived from the review findings and the experimental results describing the strengths and weaknesses of the initial prototype.
The developed prototype uses computer vision for human gesture detection and cable harness state estimation and deploys speech recognition to obtain voice commands from the user. Based on these inputs, the system controls a collaborative robot arm to manipulate the cable harness in the physical world.
In the practical experiment, the system reached an overall success rate of 0.38 with per-feature success rates ranging from 0.72 to 1.00 in an end-to-end cable-handover sequence. We consider these results sufficient for proof-of-concept purposes: while far from perfect, they prove the feasibility of our approach.
The major defects of the system include unstable grip, compromised handover accuracy, and occasional force-related hardware failures. The future development plan includes deploying force-based compliant control combined with vision-based optimization to address the hardware failures and poor handover accuracy, and integrating a customized gripper specifically designed for cable handling to improve the grip performance.