Dynamic-Structured Semantic Propagation Network

Xiaodan Liang, Hongfei Zhou, Eric Xing; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 752-761

Semantic concept hierarchy is yet under-explored for semantic segmentation due to the inefficiency and complicated optimization of incorporating structural inference into the dense prediction. This lack of modeling dependencies among concepts severely limits the generalization capability of segmentation models for open set large-scale vocabularies. Prior works thus must tune highly-specified models for each task due to the label discrepancy across datasets. In this paper, we propose a Dynamic-Structured Semantic Propagation Network (DSSPN) that builds a semantic neuron graph to explicitly incorporate the concept hierarchy into dynamic network construction, leading to an interpretable reasoning process. Each neuron for one super-class (eg food) represents the instantiated module for recognizing among fine-grained child concepts (eg editable fruit or pizza), and then its learned features flow into the child neurons (eg distinguishing between orange or apple) for hierarchical categorization in finer levels. A dense semantic-enhanced neural block propagates the learned knowledge of all ancestral neurons into each fine-grained child neuron for progressive feature evolving. During training, DSSPN performs the dynamic-structured neuron computational graph by only activating a sub-graph of neurons for each image. Another merit of such semantic explainable structure is the ability to learn a unified model concurrently on diverse datasets by selectively activating different neuron sub-graphs for each annotation at each step. Extensive experiments on four public semantic segmentation datasets (i.e. ADE20K, COCO-Stuff, Cityscape and Mapillary) demonstrate the superiority of DSSPN, and a universal segmentation model that is jointly trained on diverse datasets can surpass the common fine-tuning scheme for exploiting multi-domain knowledge.