Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

Adam Pardyl, Grzegorz Kurzejamski, Jan Olszewski, Tomasz Trzcinski, Bartosz Zielinski; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 8525-8534

Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems such as active visual exploration where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover we propose modifications to the transformer architecture and training regime which increase its elasticity. Through extensive experimentation we spotlight opportunities and challenges associated with such architecture.