TaCOS: Task-Specific Camera Optimization with Simulation

Chengyang Yan, Donald G. Dansereau; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 2052-2062

The performance of perception tasks is heavily influenced by imaging systems. However designing cameras with high task performance is costly requiring extensive camera knowledge and experimentation with physical hardware. Additionally cameras and perception tasks are mostly designed in isolation whereas recent methods that jointly design cameras and tasks have shown improved performance. Therefore we present a novel end-to-end optimization approach that co-designs cameras with specific vision tasks. This method combines derivative-free and gradient-based optimizers to support both continuous and discrete camera parameters within manufacturing constraints. We leverage recent computer graphics techniques and physical camera characteristics to simulate the cameras in virtual environments making the design process cost-effective. We validate our simulations against physical cameras and provide a procedurally generated virtual environment. Our experiments demonstrate that our method designs cameras that outperform common off-the-shelf options and more efficiently compared to the state-of-the-art approach requiring only 2 minutes to design a camera on an example experiment compared with 67 minutes for the competing method. Designed to support the development of cameras under manufacturing constraints multiple cameras and unconventional cameras we believe this approach can advance the fully automated design of cameras. Code is available on our project page at https://roboticimaging.org/Projects/TaCOS/.