PPR: Physically Plausible Reconstruction from Monocular Videos

Gengshan Yang, Shuo Yang, John Z. Zhang, Zachary Manchester, Deva Ramanan; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3914-3924

Given monocular videos, we build 3D models of articulated objects and environments whose 3D configurations satisfy dynamics and contact constraints. At its core, our method leverages differentiable physics simulation to aid visual reconstructions. We couple differentiable physics simulation with differentiable rendering via coordinate descent, which enables end-to-end optimization of, not only 3D reconstructions, but also physical system parameters from videos. We demonstrate the effectiveness of physics-informed reconstruction on monocular videos of quadruped animals and humans. It reduces reconstruction artifacts (e.g., scale ambiguity, unbalanced poses, and foot swapping) that are challenging to address by visual cues alone, and produces better foot contact estimation.