Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation

Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, Jian Yang; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4106-4115

Abstract


In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.

Related Material


[pdf] [supp]
[bibtex]
@InProceedings{Zhang_2019_CVPR,
author = {Zhang, Zhenyu and Cui, Zhen and Xu, Chunyan and Yan, Yan and Sebe, Nicu and Yang, Jian},
title = {Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}