IsoTGAN: Spatial and Geometrical Constraints At GAN And Transformer For 3D Contour Generation

Thao Nguyen Phuong, Vinh Nguyen Duy, Hidetomo Sakaino; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2024, pp. 435-452

Image generation in 2D and 3D has become an active research topic in Deep Learning. Single or multiple input images with nonorthogonal views are used for another shape and texture with different viewing angles. On the other hand, Computer-Aided Design (CAD) relies on handling-based 3D generation, i.e., isometric view images, from three orthographic view line drawings in 2D. However, since unique viewing pairs of such 2D and 3D images are required to train, SOTA models are insufficient to generate desirable images. More spatial and geometrical constraints remain undone due to less corresponding image features between images. This paper proposes IsoTGAN with a GAN-based generator with the Transformer in its generator, where three images of an object's front, side, and top view are input. The encoder is trained by the spatial and geometrical relation among three view images. A novel Gaussian Enhanced Euclidean attention mechanism and a geometryconstrained loss function are also proposed for further local image feature enhancement. Extensive experiments on the SPARE3D dataset demonstrate that the proposed IsoTGAN outperforms State-of-the-art (SOTA) models, i.e., DINO, regarding local and global image feature accuracy. This helps generate 3D isometric view images in auto-CAD system.