Latency Reduction Potential of Server-Side Command Buffers in OpenCL-Based Edge Offloading

Abstract

Compute acceleration APIs based on direct queueing of commands, such as OpenCL, have to make hard decisions whether to buffer commands internally for batched submission at the cost of potentially introducing extra latency or to submit commands to accelerators immediately, incurring a communication overhead with each command. Such latencies and overheads quickly add up in applications that repeatedly re-execute the same set of commands and becomes even more pronounced when the hardware is accessed with remote drivers such as PoCL-Remote instead of locally over a PCIe bus. For the first time in the context of compute acceleration APIs we evaluate the use of command buffers in reducing re-execution latency of command graphs in distributed OpenCL offloading. We observe performance improvements between 3x and 15x in a synthetic latency stress test with server-side command buffers to minimize client-server communication over individually submitted commands. We also include measurements of a client-side implementation of command buffers.