Perceptually Optimized Model for Near-Eye Light Field Reconstruction

Abstract

We present a learning model for reconstructing near-eye dense light field (LF) from a sparse set of multi-perspective views. The model integrates a fully-convolutional neural network and a model of the retinal image formation process optically connecting the pupil, retina and neural domains. Considering the problem of 9 × 9 near-eye LF reconstruction from the available five images, four at the corner viewpoints and one in the middle, we investigate the implications of using different loss functions in the learning process in terms of reconstruction qualities at different domains. Despite the utilized simplified retinal image formation model, the simulations reveal instructive results. In particular, combining the LF loss and the retinal focal stack loss is shown to improve the reconstruction quality of actual LF at the pupil plane, facilitating learning better features. On the other hand, concerning the retinal image quality, the model trained based on the same combination of losses is also demonstrated to produce better retinal images especially for non-Lambertian scenes, i.e., when there is monocular parallax, compared to model trained based on only retinal focal stack loss.