Perception and Planning for Dual-arm Robotic Manipulation of Multi-Deformable Linear Objects in Wire Harness Assembly

Abstract

Deformable Linear Objects (DLOs), such as cables or ropes, are widely present in both industrial and everyday activities; however, their robotic manipulation remains a significant challenge. Due to their deformability, manipulating these objects entails not only the control of their position and orientation but also their shape. Moreover, their linear geometry introduces additional challenges, such as entanglements and large deformations. These complexities become even more pronounced in scenarios involving multiple DLOs (MDLOs), where close adjacency and intricate arrangements further complicate manipulation. This research contributes to addressing these challenges, focusing on the perception and planning aspects of the robotic system. Additionally, it emphasizes practical applicability, aiming for intuitive usability and efficient integration of the proposed technologies.

To tackle the challenges associated with MDLO perception, this thesis presents a vision-based system capable of identifying the shape of each DLO under complex conditions, including occlusions, constant overlap, and cluttered backgrounds. In addition, a methodology for generating photorealistic synthetic MDLO images is introduced. This enables the creation of large and diverse datasets for effectively training Deep Learning models to perform DLO segmentation. Regarding planning, this research proposes a set of functions for dual-arm coordinated motion. These functions enable the synchronization of both arms according to different policies, allowing the control of the DLOs’ shape without external support.

To make these developments accessible to end users and facilitate their integration within a robotic system, this thesis introduces two key technologies: a task-level programming by demonstration (PbD) framework and a graphical user interface (GUI). The PbD framework allows users to configure the system for new tasks by demonstrating them. It extracts a semantic description from the demonstration and leverages predefined knowledge, such as configurable robot skills and setup descriptions, to generate a robotic program. Meanwhile, the GUI provides an intuitive interface for controlling, monitoring, and configuring ROS-based robotic systems.

Finally, the developed technologies, previously validated as standalone components, are integrated into a comprehensive robotic system and tested in a representative MDLO manipulation application: wire harness assembly. The results demonstrate the effectiveness and versatility of the proposed solutions, while also highlighting limitations that can be addressed in future work.

This doctoral thesis is a compilation-based dissertation comprising six articles, five of which have been peer-reviewed, while the sixth is currently under review.