TaxaBind: A Unified Embedding Space for Ecological Applications

Srikumar Sastry, Subash Khanal, Aayush Dhakal, Adeel Ahmad, Nathan Jacobs; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 1765-1774

Abstract


We present TaxaBind a unified embedding space for characterizing any species of interest. TaxaBind is a multimodal embedding space across six modalities: ground-level images of species geographic location satellite image text audio and environmental features useful for solving ecological problems. To learn this joint embedding space we leverage ground-level images of species as a binding modality. We propose multimodal patching a technique for effectively distilling the knowledge from various modalities into the binding modality. We construct two large datasets for pretraining: iSatNat with species images and satellite images and iSoundNat with species images and audio. Additionally we introduce TaxaBench-8k a diverse multimodal dataset with six paired modalities for evaluating deep learning models on ecological tasks. Experiments with TaxaBind demonstrate its strong zero-shot and emergent capabilities on a range of tasks including species classification cross-model retrieval and audio classification. The datasets and models are made available at https://github.com/mvrl/TaxaBind.

Related Material


[pdf] [supp] [arXiv]
[bibtex]
@InProceedings{Sastry_2025_WACV, author = {Sastry, Srikumar and Khanal, Subash and Dhakal, Aayush and Ahmad, Adeel and Jacobs, Nathan}, title = {TaxaBind: A Unified Embedding Space for Ecological Applications}, booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)}, month = {February}, year = {2025}, pages = {1765-1774} }