Multi-agent Verbal Communication enabling the execution of multiple actions through a single interaction for next generation of Human-Robot Collaboration

Abstract

In recent times, there has been a notable shift towards prioritizing human-centric approaches in industrial development. The aforementioned shift was further bolstered by the European Commission's introduction of Industry 5.0, marking the advent of a new industrial paradigm. Industry 5.0 is placing increased emphasis on integrating human-centric aspects while prioritizing resilience and sustainability. The current paradigm, Industry 4.0, prioritizes digital automation over human factors that are important to the industrial sector. As a result, socio-economic concerns have been raised that have a long-term impact on the development of the industrial sector. Thus, adopting a human-centric approach to industrial development is essential to address the concerns that arise. One such application is Human-Robot Collaboration, which combines human skills and knowledge with robot capabilities to achieve a common goal. The next generation of HRC is developed with a human-centric concept, allowing seamless and intuitive interaction between humans and robots, similar to human-human interaction. Therefore, a shift towards a more natural and intuitive method of interaction between humans and cobots is needed, where the cobots can understand human intentions and respond accordingly. Large Language Models (LLM) a groundbreaking technology in the field of Natural Language Processing, is an enabling technology of such HRC. It is capable of contextually understanding human requests and generating appropriate responses.

As such, it is crucial to introduce methods that can bridge the LLM capabilities to Human-Robot Collaboration (HRC) systems, thereby enabling next-level collaborations between humans and robots. This thesis presents a novel framework to bridge the gap between Human-Robot Collaborations and Large Language models. The framework is designed to develop a next-generation human-robot collaborative system that enables verbal communication between humans and cobots during collaborative tasks. The framework incorporates a unique hallucination filtering process, a structured prompting format, and a single instruction technique for a chain of command execution. Through extensive verification, the framework has demonstrated significantly better performance in accuracy compared to present conventional techniques. After conducting a user-based study on the physical implementation of the HRC framework, it was revealed that the workload of human operators was reduced by an estimated 38\% when performing a defined task in an HRC environment that uses this framework, compared to the conventional method of doing the same collaborative task, thus increasing the degree of Human-Centricity. In conclusion, this thesis serves as a crucial stepping stone towards the establishment of next-generation Human-Robot Collaborations, which can pave the path toward the next paradigm shift, Industry 5.0.