JOADAA: Joint Online Action Detection and Action Anticipation

Mohammed Guermal, Abid Ali, Rui Dai, François Brémond; Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 6889-6898

Action anticipation involves forecasting future actions by connecting the past events to future ones. However, this reasoning ignores the real-life hierarchy of events which is considered to be of three main parts: past, present, and future. We argue that considering these three main parts and their dependencies could improve performance. On the other hand, online action detection is the task of predicting actions in a streaming manner. In this case, one has access only to the past and present information. Therefore, in online action detection (OAD) the existing approaches miss semantics or future information which limits the performance of existing approaches. To sum up, for both of these tasks, the complete set of knowledge (past-present-future) is missing, which makes it challenging to infer action dependencies achieving good performances. To address this limitation, we propose fusing both tasks in one uniform architecture. By combining action anticipation and online action detection, our approach can cover the missing dependencies of future information in online action detection. This method, referred as JOADAA, presents a uniform model that jointly performs action anticipation and online action detection. We validate our proposed model on three challenging datasets: THUMOS, which is a sparsely annotated dataset with one action per time step, CHARADES and Multi-THUMOS, two densely annotated datasets, with more complex scenarios. JOADAA achieves SOTA results on these benchmarks for both tasks.