On The Evaluation of Neural Network Deployment Options

Abstract

It is known that machine learning inference models are already transforming computing. However, it still requires massive power usage for training multiple gigantic datasets. Furthermore, inferences can be carried out in the cloud by utilizing high-performance computing (HPC) platforms for non-time critical workflow. However, it is now commonly conducted locally on edge devices for real-time applications such as video processing, image/pattern recognition, and object detection. It is crucial that the result is generated as quickly as possible with low power and effective cost.

In this thesis, we compare multiple common neural network models' performance in terms of inference time and characterized pattern of 5 commercial edge devices: Raspberry Pi 3, Raspberry Pi 4, Jetson Nano, Jetson TX2, and Jetson AGX Xavier. These neural network models are taken mainly from 4 computer vision tasks: image classification, object detection, human pose estimation, and semantic segmentation. These models are converted from Pytorch to ONNX and TensorRT format for benchmarking. Finally, our showcase results are summed up in tables and virtualized.