Deep Learning Frameworks for 3D Perception and Assembly Cognition in Robotic Manipulation

Abstract

Robotic assembly is a multi-step, multi-object-oriented, highly constrained and structured task compared to general robotic manipulation. While the existing state of the art has achieved strong performance on isolated perception tasks such as object detection, semantic segmentation, and pose estimation, their object-centric and modular design renders them ineffective for progressive assembly tasks, where inter-object relationships and spatial and temporal dependencies play a central role. This dissertation addresses this gap by introducing a fundamental framework for enabling robotic assembly cognition through the systematic integration of perception, task-aware reasoning and actionable outputs.

The dissertation identifies several knowledge gaps across data, representation, and framework levels and contributes a series of methodological artefacts to advance the research in enabling assembly cognition. First, a simulation-based data generation pipeline is introduced to address the scarcity of assembly datasets, enabling scalable, reproducible, and methodological data acquisition. Second, a structured assembly dataset that encodes spatial, relational, temporal, and semantic attributes of progressive assemblies is introduced at the representation-level. Third, an assembly pose estimation method and finally, a learning-based approach that constitutes an assembly cognition framework for robotic manipulation is presented.

The above contributions are implemented through three peer-reviewed publications and one article pending review, and each corresponds to an artefact developed in the dissertation. The artefacts are designed, built and evaluated in simulation-based robotic software environments and empirically validated for their internal performance following a design-science-oriented research strategy. This study establishes assembly cognition as a tractable and well-defined research problem in robotic manipulation by aligning perception, task-aware reasoning, and actionable outputs within a unified framework, providing a foundation for future research on structured task manipulation.