Camera localization and 3D surface reconstruction on low-power embedded devices

Abstract

This Thesis explores the opportunities for real-time camera localization and 3D surface reconstruction on an embedded device and demonstrates one practical implementation of such. Previous implementations are analyzed, and their usability on embedded platforms is discussed. The importance of accurate and fast localization in modern and future applications is considered and taken into account in the practical implementation of the system. 3D localization and surface reconstruction can be utilized in a vast number of use cases. Some of the more prevalent use cases are its use in advanced robotics, security and military applications, geo scanning, aviation industry, and the entertainment sector. The recent advancements in extender reality and mobile devices have accelerated the adoption of high-performance localization even further. In its core, the problem of 3D localization involves inferring the position and rotation of the device both in the local case in reference to the last few frames and in the global case in reference to all of the previous frames and reconstructed 3D landmarks. Augmenting the localization problem with the reconstruction of robust 3D point clouds and a surface adds additional constraints to the requirements. Mainly, the importance of both local and global camera pose consistency is accentuated due to the triangulation of the camera-space 2D image features into world-space 3D points necessitating the fulfillment of the cheirality constraint. Additionally, deviations in the camera poses induces unwanted noise into the point surface and causes cumulative distortions in the form of the 3D surface. The implemented 3D localization and reconstruction system utilizes various simultaneous localization and mapping techniques for localizing the camera and a diverse set of structure-from-motion algorithms for reconstructing the real-world in virtual space. Concepts from edge computing and mobile robotics are used in speeding up the reconstruction and visualization workflow. On a high level, the system consists of eight (8) stages: 2D feature detection and matching, camera localization, landmark triangulation, wireless point cloud streaming, point cloud structuration, Poisson 3D surface reconstruction, and 3D visualization. The algorithms involved are examined in detail and considered from the viewpoint of embedded and power constrained devices. Appropriate measures for optimization are taken when pertinent, and the performance of the system in various scenarios is quantified by the use of performance metrics. The system is shown to be usable in real-world applications, and the obtained reconstruction results are compared against state-of-the-art open-source and academic solutions. The system is open-source under the MIT license and available on GitHub.