Real-World Photorealistic Novel View Synthesis with Consumer Devices

Abstract

Novel view synthesis is a technique employed in computer graphics to generate images, allowing an object or a scene to be observed from different perspectives. Lately, there has been a surge of interest in novel view synthesis methods. The increase in interest is thanks to the advancement in deep learning technologies. Currently, the most known successful methods are neural radiance fields and 3D Gaussian Splatting (3D-GS). Many studies have successfully optimized these methods to render real-world photorealistic scenes, although they require heavy computational power. Consequently, the practical application of these methods may be compromised due to their demanding computational power. This thesis aims to explore novel view synthesis methods such as Nerfacto, Splatfacto, 3DGS, and few-shot sparse view Gaussian Splatting (FSGS) for consumer devices. The Nerfacto and Splatfacto methods are Nerfstudio’s optimized implementations of NeRF and 3D-GS. The research involves training these methods on three different datasets and evaluating the results. Two of the datasets originate from the dataset provided by the Local Light Field Fusion (LLFF) research, and the third dataset is instead captured with a smartphone application. The experimentation phase proceeds to test Nerfacto, Splacto, 3D-GS, and FSGS methods with the datasets from the LLFF dataset. Subsequently, the testing will continue with the smartphone dataset, where only the Nerfacto and Splatfacto methods will be evaluated. The findings from the research indicate that the FSGS outperforms other methods. However, observations have shown that both Splatfacto and 3D-GS can generate photorealistic scenes given enough computing power or when appropriate optimization techniques are applied. Therefore, further studies focusing on optimizing the NVS methods will be critical for achieving photorealistic scenes with consumer devices. This advancement could potentially make it possible to explore real-world photorealistic scenes within virtual reality or augmented reality.