3D Gaussian Splatting for Real-Time Radiance Field Rendering using Insta360 Camera

Abstract

3D Gaussian splatting (GS) is a volume rendering and novel-view synthesis technique that employs 3D Gaussian functions, or splats, to accurately depict real-life and synthetic 3D scenes without utilizing neural networks. Unlike Radiance Field methods, the Gaussian splatting tech nique achieves high real-time display rates at Full-HD (1080p) resolution while eliminating significant computational costs, albeit validated only for datasets that capture small scenes (focusing on single objects). The performance and scalability of the technique on large datasets, such as entire rooms or city districts; however, remain under-explored. This study uses an Insta360 RS-One camera for creating Gaussian splatting based large scale scene (rooms or unbounded outdoor environments) representations while sustaining high real-time visual rendering quality, which is assessed through both quantitative (PSNR, SSIM and LPIPS) and qualitative metrics. The study validates the feasibility and the necessity of using the Insta360 camera over perspective cameras (e.g. iPhone 11) for creating large-scene representations using the GS method. Moreover, the study compares the Gaussian splatting method to a prominent Neural Radiance Field-based approach, namely ’nerfacto’, to highlight the Gaussian splatting method’s superior efficacy in large-scale scene reconstruction, while also offering insights into the utilization of the Insta360 camera for Radiance Field-based novel-view synthesis methods. Finally, the study integrates the use of RealityCapture, a commercial Structure-from-Motion software, into the Gaussian splatting technique as an alternative to the defacto COLMAP framework to explore potential improvements in the scene reconstruction quality, speed and computational costs. It has been shown that although perspective cameras provide sufficiently high quality captures for the Gaussian splatting method on small, bounded scene reconstructions, they fail to sustain their practicality for large-scale scenes. The captures from the limited field of vision of perspective cameras lead to distorted, blurry and inaccurate scene representations. On the other hand, the two wide-angle lenses of the Insta360 camera ensures fast and extensive large-scale scene captures that are convenient for implementing the Gaussian splatting and nerfacto methods, although the image preprocessing step introduces reduction in the resolution of dataset images, hence affecting the quality slightly. The synthesized novel views of unbounded large scenes also indicate that the Gaussian splatting method achieves superior quantitative and qualitative rendering results compared to the nerfacto method, while maintaining comparable training times. Moreover, using RealityCapture instead of COLMAP significantly fastens the image registration process for Gaussian splatting nearly by double; however posing a tradeoff by introducing minor artifacts in views that are not present with the COLMAP based GS implementation.