Conversational Generative AI Framework for Robot Control: Natural Language Understanding with Distributed Language and Vision Models : A Distributed Framework for Collaborative Manipulation with Explicit User Confirmation
Ghasemzadeh Kakroudi, Arash (2026)
2026
Master's Programme in Computing Sciences and Electrical Engineering
Informaatioteknologian ja viestinnän tiedekunta - Faculty of Information Technology and Communication Sciences
Hyväksymispäivämäärä
2026-05-04
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:tuni-202605044872
https://urn.fi/URN:NBN:fi:tuni-202605044872
Tiivistelmä
This thesis aims to investigate how LLMs and VLMs can be used to facilitate conversational manipulation of a collaborative robot arm. This would enable a non-expert user to communicate to an FR3 robot in natural language, without sacrificing transparency.
The system contains a modular ROS2 architecture where different modules process language understanding, visual object localization, coordination, motion planning, and a web-based human interface. The LLM planner operates on an edge device that converts natural language commands to intent representation, whereas the VLM module operating on a GPU workstation computes object positions from camera images. The coordinator fuses both results along with the camera-to-robot transformation matrix to produce 3D goals for a MoveIt2 motion planner. The entire pipeline operates locally without using any cloud infrastructure.
Evaluation is carried out through an experimental setup with dice in a tabletop setting. There are three types of tasks (pick, place, and handover), three scene ambiguities (single, multiple, and overlapping), and five variations for each task. Four deployable LLM-VLM configurations are tested over 180 trials, logging success rate, module-wise latency, and memory utilization. The optimal distributed configuration employs ministral-3:8b on the Jetson Orin 64 GB and Qwen2.5-VL32B on the workstation with the RTX 6000 GPU, obtaining error-prone results in this evaluation with acceptable latency, whereas lesser models yield inconsistent performance.
Results show that a full on-device, distributed multi-agent system is capable of pick-and-place and handover behavior in a controlled environment, as long as the underlying models are sufficiently accurate for the hardware. Statistical tests confirm significant performance differences between model combinations.
The system contains a modular ROS2 architecture where different modules process language understanding, visual object localization, coordination, motion planning, and a web-based human interface. The LLM planner operates on an edge device that converts natural language commands to intent representation, whereas the VLM module operating on a GPU workstation computes object positions from camera images. The coordinator fuses both results along with the camera-to-robot transformation matrix to produce 3D goals for a MoveIt2 motion planner. The entire pipeline operates locally without using any cloud infrastructure.
Evaluation is carried out through an experimental setup with dice in a tabletop setting. There are three types of tasks (pick, place, and handover), three scene ambiguities (single, multiple, and overlapping), and five variations for each task. Four deployable LLM-VLM configurations are tested over 180 trials, logging success rate, module-wise latency, and memory utilization. The optimal distributed configuration employs ministral-3:8b on the Jetson Orin 64 GB and Qwen2.5-VL32B on the workstation with the RTX 6000 GPU, obtaining error-prone results in this evaluation with acceptable latency, whereas lesser models yield inconsistent performance.
Results show that a full on-device, distributed multi-agent system is capable of pick-and-place and handover behavior in a controlled environment, as long as the underlying models are sufficiently accurate for the hardware. Statistical tests confirm significant performance differences between model combinations.