End-to-end Surface Reconstruction For Touching Trajectories

Jiarui Liu, Yuanpei Zhang, Zhuojun Zou, Jie Hao; Proceedings of the Asian Conference on Computer Vision (ACCV), 2022, pp. 2354-2370

Whereas vision based 3D reconstruction strategies have progressed substantially with the abundance of visual data and emerging machine-learning tools, there are as yet no equivalent work or datasets with which to probe the use of the touching information. Unlike vision data organized in regularly arranged pixels or point clouds evenly distributed in space, touching trajectories are composed of continues basic lines, which brings more sparsity and ambiguity. In this paper we address this problem by proposing the first end-to-end haptic reconstruction network, which takes any arbitrary touching trajectory as input, learns an implicit representation of the underling shape and output a watertight triangle surface. It is composed of three modules, namely trajectory feature extraction, 3D feature interpolation, as well as implicit surface validation. Our key insight is that formulating the haptic reconstruction process into an implicit surface learning problem not only brings the ability to reconstruct shapes, but also improves the fitting ability of the network in small datasets. To tackle the sparsity of the trajectories, we use a spatial gridding operator to assign features of touching trajectories into grids. A surface validation module is used to tackle the dilemma of computing resources and calculation accuracy. We also build the first touching trajectory dataset, formulating touching process under the guide of Gaussian Process. We demonstrate that our method performs favorably against other methods both in qualitive and quantitative way. Insights from the tactile signatures of the touching will aid the future design of virtual-reality and human-robot interactions.