Physically-Plausible Illumination Distribution Estimation

Egor Ershov, Vasily Tesalin, Ivan Ermakov, Michael S. Brown; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 12928-12936

A camera's auto-white-balance (AWB) module operates under the assumption that there is a single dominant illumination in a captured scene. AWB methods estimate an image's dominant illumination and use it as the target "white point" for correction. However, in natural scenes, there are often many light sources present. We performed a user study that revealed that non-dominant illuminations often produce visually pleasing white-balanced images and, in some cases, are even preferred over the dominant illumination. Motivated by this observation, we revisit AWB to predict a distribution of plausible illuminations for use in white balance. As part of this effort, we extend the Cube++ illumination estimation dataset to provide ground truth illumination distributions per image. Using this new ground truth data, we describe how to train a lightweight neural network method to predict the scene's illumination distribution. We describe how our idea can be used with existing image formats by embedding the estimated distribution in the RAW image to enable users to generate visually plausible white-balance images.