Analyzing the Internals of Neural Radiance Fields

Lukas Radl, Andreas Kurz, Michael Steiner, Markus Steinberger; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 2822-2831

Modern Neural Radiance Fields (NeRFs) learn a mapping from position to volumetric density leveraging proposal network samplers. In contrast to the coarse-to-fine sampling approach with two NeRFs this offers significant potential for acceleration using lower network capacity. Given that NeRFs utilize most of their network capacity to estimate radiance they could store valuable density information in their parameters or their deep features. To investigate this proposition we take one step back and analyze large trained ReLU-MLPs used in coarse-to-fine sampling. Building on our novel activation visualization method we find that trained NeRFs Mip-NeRFs and proposal network samplers map samples with high density to local minima along a ray in activation feature space. We show how these large MLPs can be accelerated by transforming intermediate activations to a weight estimate without any modifications to the training protocol or the network architecture. With our approach we can reduce the computational requirements of trained NeRFs by up to 50% with only a slight hit in rendering quality. Extensive experimental evaluation on a variety of datasets and architectures demonstrates the effectiveness of our approach. Consequently our methodology provides valuable insight into the inner workings of NeRFs.