Novel view synthesis of 3D Scene with transparent and reflective surface : An experimental dataset to challenge current novel view synthesis methods

Abstract

The development of technologies in the metaverse has never been more significant and exciting than in the past couple of years. Applications in VR/AR or creative works have been growing nonstop. It is expanding humans' capability to replicate real life with our creativity into the metaverse world, encouraging development and the discovery of the best method. However, with constant development, there is a vast difference between the ideal experiments during testing and the harsh real-life conditions one might encounter when utilizing the technologies. When one tries to capture an actual scene to reconstruct it in the 3D world, it would be good to assume that the capturing process is not perfect and could potentially have many inconsistent problems. This situation happens in most cases regarding public use of the technology. To address this difference, different methods that are in development should be able to test with more diverse and realistic datasets. Thus, this thesis aims to propose a small dataset that consists of common inconsistencies and the evaluation of the said dataset on current popular methods in view synthesis and view reconstruction. To construct such a dataset, requirements on the "problems" the dataset should have were listed clearly, and thus, the final dataset was collected and organized accordingly. The testing methods currently include the two most popular methods: NeRF and 3D Gaussian Splatting. Additionally, the Gaussian Shader method, developed based on 3D Gaussian Splatting, was also studied and tested as an example of a method that targets solely one "problem," correctly rendering reflective surfaces without the surrounding background information. Throughout the testing of the methods, we have indeed observed the efficiency of the 3D Gaussian Splatting method, which is superior to the NeRF method, explaining the popularity of the technique at the current time. Meanwhile, the results from the Gaussian Shader method have proven its true characteristic and efficiency when it comes to either solely reflective surface scenes or failure when it comes to more diverse and robustness-required scenes. Due to the time limit, we only tested a few such methods. However, the results have indicated that the dataset has served its purpose and can be further developed into a more complex dataset based on this current simple protocol.