Localised-NeRF: Specular Highlights and Colour Gradient Localising in NeRF

Dharmendra Selvaratnam, Dena Bazazian; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2791-2801

Neural Radiance Field (NeRF) based systems predominantly operate within the RGB (Red Green and Blue) space; however the distinctive capability of the HSV (Hue Saturation and Value) space to discern between specular and diffuse regions is seldom utilised in the literature. We introduce Localised-NeRF which projects the queried pixel point onto multiple training images to obtain a multi-view feature representation on HSV space and gradient space to obtain important features that can be used to synthesise novel view colour. This integration is pivotal in identifying specular highlights within scenes thereby enriching the model's understanding of specular changes as the viewing angle alters. Our proposed Localised-NeRF model uses an attention-driven approach that not only maintains local view direction consistency but also leverages image-based features namely the HSV colour space and colour gradients. These features serve as effective indirect priors for both the training and testing phases to predict the diffuse and specular colour. Our model exhibits competitive performance with prior NeRF-based models as demonstrated on the Shiny Blender and Synthetic datasets. The code of Localised-NeRF is publicly available (https://github.com/Dharmendra04/Localised-NeRF).