Sharing is Caring: Concurrent Interactive Segmentation and Model Training Using a Joint Model

Ivan Mikhailov, Benoit Chauveau, Nicolas Bourdel, Adrien Bartoli; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 2432-2441

The performance of neural predictors depends on the size and composition of the training dataset. However, annotating data is expensive. Efficient annotation systems usually feature a neural annotation predictor whose result can be edited by the expert using classical tools. Existing systems train the annotation predictor from an initial small subset of data annotated by classical tools and then freeze it for the rest of the annotation process. This is suboptimal as the annotation predictor does not benefit from the new annotations as the annotation process progresses. We propose a framework called Single Active Interactive Model (SAIM), which integrates the three steps of data selection, annotation and training into a single architecture. This is made possible by three key properties of SAIM in contrast with existing work: 1) SAIM uses a deep interactive predictor; hence the classical tools are not required and the annotation predictor can be pre-trained with limited data to produce quality annotations; 2) SAIM uses a single model shared between the three steps, hence the model is deployable and the annotation predictor improves as annotation progresses; 3) SAIM uses active learning to maximise the impact of each annotation on the predictor performance, making the model rapidly improve. We evaluated SAIM by emulating annotation scenarios on fully-labelled segmentation datasets. For a complex female pelvis MRI dataset, pre-training SAIM on 15% of data and annotating the whole dataset achieves 73.4% IoU with 6.3 hours of annotation time, against 75.8% IoU for complete manual annotation, requiring 40.0 hours. We also applied SAIM to a real-world case of very large MRI dataset (AMOS) segmentation, which cannot be feasibly annotated otherwise.