A Neural Video Codec With Spatial Rate-Distortion Control

Noor Fathima, Jens Petersen, Guillaume Sautière, Auke Wiggers, Reza Pourreza; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp. 5365-5374

Neural video compression algorithms are nearly competitive with hand-crafted codecs in terms of rate-distortion performance and subjective quality. However, many neural codecs are inflexible black boxes, and give users little to no control over the reconstruction quality and bitrate. In this work, we present a flexible neural video codec that combines ideas from variable-bitrate codecs and region-of-interest-based coding. By conditioning our model on a global rate-distortion tradeoff parameter and a region-of-interest (ROI) mask, we obtain fine control over the per-frame bitrate and the reconstruction quality in the ROI. The resulting codec enables practical use cases such as coding under bitrate constraints with fixed ROI quality, while taking a negligible hit in overall rate-distortion performance. We find that our codec is best utilized when the sequence contains complex motion, such as scenes with camera panning or sports videos, where we substantially outperform non-ROI codecs in the region of interest with BD-rate savings exceeding 60% in some cases.