MatSpectNet: Material Segmentation Network with Domain-Aware and Physically-Constrained Hyperspectral Reconstruction

Yuwen Heng, Yihong Wu, Srinandan Dasmahapatra, Hansung Kim; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 8079-8089

Achieving accurate material segmentation for 3-channel RGB images is challenging due to the considerable variation in material appearance. Hyperspectral images which are sets of spectral measurements sampled at multiple wavelengths theoretically offer distinct information for material identification as variations in the intensity of electromagnetic radiation reflected by a surface depend on the material composition of a scene. However existing hyperspectral datasets are impoverished in terms of the number of images and material categories for the dense material segmentation task and collecting and annotating hyperspectral images with a spectral camera is prohibitively expensive. To address this we propose a novel model the MatSpectNet to segment materials with recovered hyperspectral images from RGB images. The network leverages the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images and employs the domain adaptation method to generalise the hyperspectral reconstruction capability from a spectral recovery dataset to material segmentation datasets. The reconstructed hyperspectral images are further filtered using learnt response curves and enhanced with human perception. The performance of MatSpectNet is evaluated on the LMD dataset as well as the OpenSurfaces dataset. Our experiments demonstrate that MatSpectNet attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication. Additional experiments and the project code are published on https://github.com/heng-yuwen/MatSpectNet.