RAW Bayer Domain Image Alignment

Abstract

Every day millions of photographs are captured with handheld mobile devices. Depending on the capturing conditions, multiple images may be captured in order to obtain more information on the scene in question. These multiple images are captured sequentially and called raw images. As a result, depending on the scene dynamics and due to the camera shake (while shooting), raw images will have pixel shifts. Because of that reason, during raw image capturing, images may be slightly misaligned, resulting in ghosting artifacts in the fused image. Raw images need to be aligned in order to achieve an effective fusion. Optical flow plays an important role in image alignment as it can estimate any object motion in the scene while also capturing global camera motion. It is a flow vector that determines how much each pixel of the reference frame is shifted in the moving frame. Then, using that flow vector, the moving frame can be warped to the reference frame to reduce misalignment between the reference and moving frames. The purpose of this thesis is to evaluate the performance of optical flow algorithms to objectively analyze optical flow accuracy and to assess the amount of ghosting artifacts in the fused image in raw Bayer domain. Therefore, the input data is expected to be close to a raw image. However, publicly available optical flow datasets are far from this requirement. For this reason, a new in-house raw image containing dataset has been created. In this thesis, both conventional and deep learning-based optical flow methods were explored. The performance of the optical flow methods was evaluated on both the publicly available datasets and the in-house dataset. Based on the results obtained, both deep learning-based and conventional methods perform well when the motion between consecutive images is relatively small. However, with the increasing amount of motion, the motion estimation is more error-prone and does not perform as desired. Furthermore, the performance of optical flow algorithms was tested on different levels of noise. It has been observed that noise has a negative effect on optical flow performance. Moreover, the findings indicated that optical flow algorithms achieved remarkable overall quality improvement after applying Black Level Correction (BLC), Lens Shading Correction (LSC), White Balance (WB), and Opto-Electronic Transfer Function (OETF) to images in raw Bayer domain. Also, deep learning-based algorithms operate on RGB images. However, raw Bayer images have four channels (RGGB). Therefore, two different demosaicing algorithms were tried to reconstruct RGB images from Bayer patterns, the one green of two green channels skipping mode and bilinear interpolation. The results have shown that the alignment of the RGB images constructed by bilinear interpolation produced higher sub-pixel accuracy than the skipping mode.