CoGS: Controllable Gaussian Splatting

Heng Yu, Joel Julin, Zoltán A. Milacski, Koichiro Niinuma, László A. Jeni; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 21624-21633

Capturing and re-animating the 3D structure of articulated objects present significant barriers. On one hand methods requiring extensively calibrated multi-view setups are prohibitively complex and resource-intensive limiting their practical applicability. On the other hand while single-camera Neural Radiance Fields (NeRFs) offer a more streamlined approach they have excessive training and rendering costs. 3D Gaussian Splatting would be a suitable alternative but for two reasons. Firstly existing methods for 3D dynamic Gaussians require synchronized multi-view cameras and secondly the lack of controllability in dynamic scenarios. We present CoGS a method for Controllable Gaussian Splatting that enables the direct manipulation of scene elements offering real-time control of dynamic scenes without the prerequisite of pre-computing control signals. We evaluated CoGS using both synthetic and real-world datasets that include dynamic objects that differ in degree of difficulty. In our evaluations CoGS consistently outperformed existing dynamic and controllable neural representations in terms of visual fidelity.