From Leaderboard to Operations: DIVA Transition Experiences

Bharadwaj Ravichandran, Roderic Collins, Keith Fieldhouse, Kellie Corona, Anthony Hoogs; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2022, pp. 143-150

The IARPA Deep Intermodal Video Analytics (DIVA) program has sponsored the development of systems that detect and recognize activities in security video. During the period from September 2017 to March 2021, the development and evaluation of these systems was focused on optimizing accuracy, embodied in quantified metrics, against a large but relatively static corpus of video collected and annotated by the program. This focus was aided by various software engineering decisions collaboratively reached by the program performers and Test & Evaluation (T&E) team, which established a common software framework enabling ongoing quantitative evaluation via software submissions to a leaderboard. While continuing to support the leaderboard, in March 2021 the program began efforts, still in progress, to transition capabilities developed on DIVA from the research environment to operational evaluation and deployment. As an operational system is a different use case than a research environment, it is not surprising that design decisions favoring the former will not always align with the latter. This paper discusses our work to transition DIVA systems into an operational setting, particularly identifying and resolving conflicts between the evaluation framework and operational requirements. We describe transition efforts to date, propose future work, and conclude with lessons learned from the overall transition effort.