Learned video compression with intra-guided enhancement and implicit motion information

Abstract

Although learned approaches to video compression have been proposed with promising results, hand-engineered video codecs are still unbeaten. On the other hand, learned image compression has already surpassed traditional image codecs. In this paper, we propose a learned video compression system that mimics part of the pipeline of traditional codecs while leveraging learned image compression. It comprises two main modules: a learned intra-frame compression module, and a learned inter-frame compression module that is conditioned on intra-coded frames. These modules use separate learned probability models for entropy coding. The intra-frame codec uses a variant of nonlocal attention layers. Regarding the inter-frame codec, we propose an implicit motion information mechanism, and an enhancement of the inter-frame predictions by leveraging the high quality information of intra-coded frames. On the learned probability model side, we propose to use the reference frames as additional conditioning information. We used this system as our submitted entry for the 2021 Challenge on Learned Image Compression (CLIC). In our experiments, we show the effectiveness of our system and its components via a set of ablation studies.