Design and implementation of a reinforcement learning policy deployment pipeline for an excavator

Abstract

Reinforcement learning is seen as one of the most promising methods to automate an excavator. Reinforcement learning policies are trained in simulations to lower the cost, save time, and avoid safety concerns. The goal of this thesis is to implement a pipeline to deploy a simulation-trained reinforcement learning policy on a real excavator.

The background section provides an introduction to excavators and excavation automation. Then, it introduces how certain kinematic parameters can be computed that are required for the policy, and basic principles behind reinforcement learning. Lastly, simulation-to-real transfer is introduced and Robot Operating System 2 as well.

The implemented pipeline consists of two ROS 2 nodes. The first node is an observation builder that constructs the observation for the policy from the excavator’s sensor data. The second node is policy execution that executes the simulation-trained policy and sends the action commands to the excavator’s controller. Configuration for the observation vector is read from a YAML file, which allows flexible modifications to the constructed vector. Policy is executed via ONNX runtime.

The implementation was tested on a real excavator together with a goal-reaching policy. The deployment pipeline was able to construct the observation, execute the policy and send action commands to the excavator’s controller. However, the targeted goal-reaching behaviour was not achieved in this test. During the test, discrepancies occurred in computed observations which limit the pipeline’s reliability. Currently, the pipeline is suitable for policies that do not require computed observations.