% CVPR 2023 Paper Template
% based on the CVPR template provided by Ming-Ming Cheng (https://github.com/MCG-NKU/CVPR_Template)
% modified and extended by Stefan Roth (stefan.roth@NOSPAMtu-darmstadt.de)

\documentclass[10pt,twocolumn,letterpaper]{article}

%%%%%%%%% PAPER TYPE  - PLEASE UPDATE FOR FINAL VERSION
%\usepackage[review]{cvpr}      % To produce the REVIEW version
\usepackage{cvpr}              % To produce the CAMERA-READY version
%\usepackage[pagenumbers]{cvpr} % To force page numbers, e.g. for an arXiv version

% Include other packages here, before hyperref.
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{multirow}


% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues, e.g. with the
% file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete
% ReviewTempalte.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you
%  should be clear).
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}


% Support for easy cross-referencing
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}


%%%%%%%%% PAPER ID  - PLEASE UPDATE
\def\cvprPaperID{*****} % *** Enter the CVPR Paper ID here
\def\confName{CVPR}
\def\confYear{2023}


\begin{document}

%%%%%%%%% TITLE - PLEASE UPDATE
\title{An Actor-centric Causality Graph for Asynchronous Temporal Inference in Group Activity}

\author{Zhao Xie, Tian Gao, Kewei Wu\thanks{Corresponding author.}, Jiao Chang\\
Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology\\
School of Computer Science and Information Engineering, Hefei University of Technology\\
{\tt\small xiezhao@hfut.edu.cn; gaotian@mail.hfut.edu.cn; wukewei@hfut.edu.cn; jiaochang@mail.hfut.edu.cn}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
%\and
%Second Author\\
%Institution2\\
%First line of institution2 address\\
%{\tt\small secondauthor@i2.org}
}
\maketitle

%%%%%%%%% ABSTRACT
\begin{abstract}
The causality relation modeling remains a challenging task for group activity recognition.
The causality relations describe the influence on the centric actor (effect actor) from its correlative actors (cause actors).
Most existing graph models focus on learning the actor relation with synchronous temporal features, which is insufficient to deal with the causality relation with asynchronous temporal features.
In this paper, we propose an Actor-Centric Causality Graph Model, which learns the asynchronous temporal causality relation with three modules, i.e., an asynchronous temporal causality relation detection module, a causality feature fusion module, and a causality relation graph inference module.
First, given a centric actor and its correlative actor, we analyze their influences to detect causality relation.
We estimate the self influence of the centric actor with self regression.
We estimate the correlative influence from the correlative actor to the centric actor with correlative regression, which uses asynchronous features at different timestamps.
Second, we synchronize the two action features by estimating the temporal delay between the cause action and the effect action.
The synchronized features are used to enhance the feature of the effect action with a channel-wise fusion.
Third, we describe the nodes (actors) with causality features and learn the edges by fusing the causality relation with the appearance relation and distance relation.
The causality relation graph inference provides crucial features of effect action, which are complementary to the base model using synchronous relation inference.
Experiments show that our method achieves state-of-the-art performance on the Volleyball dataset and Collective Activity dataset.
\end{abstract}

%%%%%%%%% BODY TEXT
\section{Introduction}

\begin{figure}[t] 
\centering  
\includegraphics[width=0.5\textwidth]{figs/mot.png} 
\caption{Illustration of the causality relation. 
(a) Asynchronous causality relation.
(b) Synchronous temporal relation graph.
{\color{black}(c) Asynchronous temporal causality relation detection and fusion.}
(d) Asynchronous temporal causality relation graph learns the influences of two actors to detect causality relation.
In this work, we enforce the graph model with asynchronous causality relation by analyzing the actors' influences.
}
\label{mot} 
\end{figure}

Group activity recognition is a challenging task to identify the group activity by analyzing the actors that perform different actions.
It has been widely used in many applications in video surveillance, social interaction analysis, and sports analysis \cite{hdtm,strap,saacrf,fcar}.
Unlike individual action recognition, group activity recognition learns the relation between actors to infer the group activity.
The relation between two actors can be explained as cause-effect relation (denoted as causality relation), in which the action of one actor (denoted as the cause actor) impacts the action of another actor (denoted as the effect actor).
The two actors impact each other individually resulting in different causality relations.
As the effect action performs after the cause action with a temporal delay, the asynchronous temporal features of the two actors hinder causality relation learning.
Existing methods describe the relation with the appearance feature \cite{hrn} and the position feature \cite{arg,pdar}.
The above methods merely learn the spatial relation in each frame, which neglect to describe the relation with the temporal dynamics in the frame sequence.
Some methods describe the relation with the temporal feature learned by RNN \cite{glil} and Transformer network \cite{gf}.
The existing methods always learn the relation at the same timestamp with the synchronous temporal feature, and neglect to describe the influences of two actors, who have asynchronous temporal relation.
Therefore, it is still challenging to capture the relation with asynchronous temporal features for group activity recognition.

As the causality relation is asynchronous, we decompose our graph model with asynchronous causality relation into two sub-tasks illustrated in Figure \ref{mot}b: 
(1) the causality relation detection task by analyzing the influences of two actors with their asynchronous features, and (2) the asynchronous causality feature fusion task by integrating the feature of the effect actor with the synchronized feature of the cause actor.

Figure \ref{mot} shows the group activity prediction influenced by different actors.
{\color{black}{(1) In Figure \ref{mot}a, with a temporal delay after the cause action performs (digging), the cause actor changes to the action "falling", and the effect actor changes to the action "jumping".}}
Actors change their states with a temporal delay in an asynchronous way, which hinders cause-effect (causality) relation learning.
(2) As shown in Figure \ref{mot}b, the traditional method addresses the group activity using a synchronous temporal relation graph, which contains a large number of irrelevant relations.
The synchronous graph is hard to detect the cause-effect relation with the synchronous features at a single timestamp.
Without learning the asynchronous causality relation, the model can not explain why the effect actor jumps.
For example, it uses the falling cause actor to mispredict group activity as the "Right pass". 
(3) {\color{black}{To detect the causality relations between the centric actor and its correlative actor, we focus on learning the influences with their asynchronous features.}}
When the centric actor is affected by the correlative actor, the influence of two actors is larger than the influence of the centric actor itself.
The influences of two actors can detect the causality relation from the correlative actor to the centric actor.
As shown in Figure \ref{mot}c, after the correlative actor moves, the centric actor jumps with the temporal delay of one frame (+1).
Then, the causality relation is used to enhance the centric actor features by fusing two actors' features.
(4) When we learn asynchronous causality relations to form a causality relation graph, which can select the relevant edges and enhance node features.
In Figure \ref{mot}d, the causality inference process finds the relation from the moving actor to the jumping actor, and helps to explain the actor jumps for setting the volleyball.
The causality relation analyzes the influences learned with the asynchronous temporal features, which is complementary to the relation learned with synchronous temporal features.
The framework by integrating two relation graphs can successfully predict the group activity as the “Right set”.



In this paper, we propose an Actor-Centric Causality Graph (ACCG) Model to detect the asynchronous causality relation for group activity recognition.
Figure \ref{overview} shows the overview of the proposed model.
The model consists of three modules, i.e., an asynchronous temporal causality relation detection module, a causality feature fusion module, and a causality relation inference module.
First, we detect the causality relation between the centric actor and its correlative actor by analyzing the self influence and the correlative influence.
We learn the self influence with self regression, and learn the correlative influence with correlative regression.
We extend the correlative regression with asynchronous features of the correlative actor, which helps to learn the asynchronous causality relation by analyzing the influences of the two actors.
Second, the temporal delay of the causality relation is estimated to synchronize two action features.
We integrate them with a channel-wise fusion to learn the causality feature of the effect actor.
Third, we describe the actors (nodes) with asynchronous causality features, and describe the edges with the causality relation for graph inference.
The causality relation inference provides the crucial features of actors, which are complementary to synchronous relation inference.
We apply the base model to learn the synchronous relation inference, and add two relation inferences to enhance the group relation learning.


Our contributions are summarized as follows:

(1) We propose an Actor-Centric Causality Graph Model, which detects the asynchronous causality relation by analyzing the influences of two actors at different timestamps.
We design the self regression to estimate the self influence of the centric actor.
We design the corelative regression with the asynchronous features of two actors to estimate their correlative influence.

(2) We design a causality feature fusion to enhance the feature of the centric actor by integrating it with the synchronized feature of its correlative actor.
The synchronized feature is generated by estimating the temporal delay between the asynchronous features of two actors.

(3) Our Actor-Centric Causality Graph Model learns the asynchronous relation, which is complementary to synchronous relation learning.
Our framework integrates two relations and achieves state-of-the-art performance on the Volleyball dataset and Collective Activity dataset.

\begin{figure*}[t] 
\centering  
\includegraphics[width=0.84\textwidth]{figs/mot-tech.png} 
\caption{{\color{black}{The overview of the actor-centric causality graph model.}}
The base model learns the relation with synchronous features at each timestamp.
The actor-centric causality graph is proposed to analyze the influence of two actors with asynchronous features, which can learn the causality relation in the asynchronous temporal causality relation detection module.
The causality feature fusion model enhances the centric action with synchronized correlative action features.
The causality relation graph inference module learns the contextual feature with causality relation.
The framework combines the causality relation graph and the base graph for better graph relation learning.
}
\label{overview} 
\end{figure*}

\section{Related Work}

\textbf{Group Activity Recognition.}
Group activity recognition learns the group feature by organizing the actor feature with relation.
Existing methods use graph models to learn the relation between actors with appearance features \cite{crm,hrn} and position features \cite{pdar,arg}. 
Some methods learn the relation adaptively by introducing self-attention \cite{gaim,dadnet} and multi-head attention \cite{at,gf}.
Some methods learn the relation by selecting the key actors \cite{mlir}.
The above methods learn the relation with spatial features.
Some methods learn the relation with temporal features \cite{prl,stagnet,cern,glil,ccglstm,proposal,iterative,context}.
These methods use the feature at the same timestamp, and neglect to learn the relation by analyzing the asynchronous temporal features.

\textbf{Granger Causality Test.}
The causality test describes the cause-effect relation of two things \cite{acre}.
Some methods learn the cause-effect relation with and-or graph \cite{pc}, and bayesian inference \cite{tbct}.
The Granger causality test learns the cause-effect relation by analyzing the feature residual estimated with temporal feature regression \cite{icr} and extends to lagged temporal analysis \cite{dgc,torvam}, reversal temporal analysis\cite{tgc}, and selected feature analysis\cite{gcdahp}. 
The Granger causality test has not been used to analyze the relation between two actors for group activity recognition.

\textbf{Asynchronous Temporal Modeling.}
The asynchronous features have been studied for the temporal feature integration \cite{astmn,aia}.
The asynchronous features can be fused with multiple temporal features \cite{atf} and multiple modality features, including appearance feature and motion feature \cite{af}.
The asynchronous features have been exploited with spatial-temporal asynchronous normalization \cite{stan}.
Unlike the above methods, we use asynchronous features to model the feature reconstruction of two actors in the group activity.
The feature reconstruction can learn the influences of them, and can be analyzed with the Granger causality test to detect causality relation.

\section{Base Group Activity Recognition Model}
Most existing methods predict the group activity with three main stages as shown in Figure \ref{overview} \cite{arg,din,gf,saacrf}.
(1) The feature extraction stage extracts features of the frame sequence using the pre-trained backbone model.
(2) The graph modeling stage learns the relation to provide contextual features of the actors.
(3) The prediction stage generates the actor action label and the group activity label and computes the prediction loss for training.
In the following, we review the common practices of the graph relation model, which is used as our base model.

\subsection{Feature Extraction}
Following \cite{arg, gf}, we use Inception-v3 to extract a feature map for a video sequence, and use RoIAlign to extract the features of the actor's bounding boxes. 
After that, an FC layer is performed on the aligned features to get the feature map for actors $X \in  \mathbb{R}^{T \times N_d\times D}$. $T$, $N_d$, and $ D $ denote frame number, actor number, and feature dimension. 

Following \cite{gf}, the group feature extractor uses the backbone model to generate the feature map of each frame.
2D convolution is applied to learn the group feature sequence $X_g \in \mathbb{R}^{T \times K \times H \times W}$.
$K$, $H$, and $W$ denote the token number, the height, and the width of the group feature.
The group encoder reshapes the feature into a flattened feature and uses a softmax operation to generate the spatial attention matrix.
2D convolution is used to project the group feature into $D$ channels, which is used to learn $K$ tokens by computing the weighted summation of every pixel with spatial attention.
Another average pooling is used to get the group token $X_G \in \mathbb{R}^{T \times D}$.

\subsection{Spatial-Temporal Graph}
Following \cite{gf}, the base synchronous relation can be learned with a temporal encoder and a spatial encoder in a stacked manner.
The temporal encoder adopts a transformer-based model to learn the temporal relation of each actor.
The input actor feature sequence is split to get the feature sequence of each actor $X_i \in \mathbb{R}^{T \times D}$, where $i$ is the actor index.
The transformer-based temporal feature is estimated as:
\begin{equation}
	\left\{ {\begin{array}{*{20}{ll}}
	Q_i=X_i W_Q, K_i=X_i W_K, V_i=X_i W_V, \\
	V'_i=softmax(\frac{Q_i K_i^T}{\sqrt{D}})V_i+V_i\\
	V''_i=FFN(V'_i)\\
	\end{array}} \right.
\end{equation}
where $W_Q$, $W_K$, $W_V$ are learnable parameters shaped as $D \times D$. 
FFN is the feed-forward network in the canonical Transformer.
The features of all actors are packed together to the temporal features $V_T \in \mathbb{R}^{N_d \times T \times D}$.

The spatial encoder learns the spatial relation between actors.
The spatial encoder follows the operation of the temporal encoder.
The difference with the above temporal encoder is that the spatial encoder uses the features at each timestamp, which is split from the output of the temporal encoder.
We pack the output of the spatial encoder to get the features of all frames $V_S \in \mathbb{R}^{T \times N_d \times D}$.


\subsection{Group Activity Prediction and Training Losses}
The base model predicts the actor action scores and group activity scores.
For actor action recognition, a classifier with two FC layers takes the learned actor features as input to predict each actor action score.
For group activity recognition, a group decoder \cite{gf} is applied to predict the group activity.
The group decoder takes the group token as the group query. 
The decoder takes the group pooling features from individual features as the group key.
The updated group query is learned by summarizing the group query with the overall context from the group pooling features.
The group classifier uses two FC layers to predict the group activity scores.


Given the ground truth labels for actor action and group activity, the loss function considers the actor action recognition loss, the group activity recognition loss, and the relation contrastive loss following \cite{saacrf}.
The contrastive loss compares the feature similarity between the nodes in the same relation graph and the similarity between the nodes in different relation graphs.
The contrastive loss encourages the diversity of relation learning.



\subsection{Discussion}
The base model uses a transformer-based temporal encoder, which can enhance the temporal features of actors.
The base model learns the spatial relation with the synchronous temporal features at the same timestamp, and neglects to analyze the influence of two actors with asynchronous features.
The main focus of our work is to analyze the influence of two actors with asynchronous features, which helps to detect the asynchronous causality relation for relation learning, as described in the following section.

\section{Actor-Centric Causality Graph Model}
Figure \ref{overview} shows the overview of our proposed methods by embedding the actor-centric causality graph in the graph modeling stage.
The graph learns the asynchronous causality relation, which is complementary to the relation with synchronous temporal features.
We represent the actor-centric causality graph as $G^{cau} =\{V^{cau}, E^{cau}\}$.
The causality relations are learned by analyzing the influences of two actors in the Asynchronous Temporal Causality Detection Module.
The nodes $V^{cau}$ describe the causality fused features of actors, which are learned with the Causality Features Fusion Module.
The edges $E^{cau}$ detail the causality relations with appearance relations and distance relations, which are used for graph relation reasoning in the causality relation inference module.


\subsection{Asynchronous Temporal Causality Relation Detection Module}
The causality relation indicates the cause actor and effect actor.
We introduce feature reconstruction to learn the reconstruction residual, which can be used to estimate the action influence.
The effect actor takes a large influence from the cause actor, and its action has a large residual of feature reconstruction with its historical features.
When the feature reconstruction considers the historical features of the cause action and effect action, the residual of effect action reconstruction becomes small.
To detect the causality relation, we select the centric actor and correlative actor.
To learn the self influence, we design self regression to model the feature reconstruction with the historical features of the centric actor.
To learn the correlative influence, we design corelative regression to model the feature reconstruction with the historical features of the two actors.
The large difference between the two influences detects a strong causality relation between the two actors.


\textbf{Self Influence Estimation.}
We estimate the self influence with the residual between the frame feature and the reconstructed feature, which is reconstructed with the historical feature of the actor itself.
Given the feature sequence $X_i \in \mathbb{R}^{T \times D}$,
We design self regression to reconstruct the current frame $\hat{x}_{k}^i$ using the historical feature bank with the time window $[k-m, k-1]$.
$m$ is the window size.
{\color{black}{The self influence $ssr_{}^{i}$ is estimated with the sum of squares residual (SSR) as:}}
\begin{small}
\begin{equation}
\left\{\begin{split}
\hat{x}_{k}^{i} &=\sum_{r=k-m}^{k-1} \omega_{r}^{i}  x_{r}^{i} + b_{}^{i}\\
ssr_{}^{i} &= \sum_{k} \left\|x_k^{i}-\hat{x}_{k}^{i}\right\|_{2}^{2}
    \end{split}
\right.
\end{equation}
\end{small}
where $\omega_{r}^{i}$, $b_{}^{i}$ are the parameters in self regression. 
The parameters are learned with the sum of the squared reconstruction error $\mathcal{L}_{self}=ssr_{}^{i}$. 


\textbf{Correlative Influence Estimation.}
We estimate the correlative influence using the construction model with two actors' features.
We design the correlative regression for feature construction.
We extend it to asynchronous correlative regression by temporally shifting the features of the correlative features with the temporal delay.
The temporal delay explains the effect action occurs after the cause action.
Given the correlative actor features $X_j$ and the temporal delay $delay$, the asynchronous historical time window of the correlative actor is $[k-delay-m, k-delay-1]$.
The asynchronous historical features of two actors are used to estimate the reconstruction feature of the centric actor $\hat{x}^{j \to i}_{k}$.
{\color{black}{The correlative influence $srr_{}^{j \to i}$ is estimated with the sum of squares residual (SSR) as:}}
\begin{small}
\begin{equation}
\left\{
	\begin{split}
\!\hat{x}^{j \rightarrow i }_{k}\! &\!=\! \sum_{r=k-m}^{k-1} \omega_{r}^{j \to i}  x^i_{r} 
\!+\! \! \! \! \sum_{r'=k-delay-m}^{k-delay-1} \! \! \! \! \omega_{r'}^{j \to i}  x^j_{r'} + b_{}^{j \to i}\\
ssr_{}^{j \to i} &= \sum_{k}\| x^i_{k} - \hat{x}^{j \rightarrow i}_{k}\|_{2}^{2}\\
    \end{split}
\right.
\end{equation}
\end{small}
where $\omega_{r}^{j \to i}$, $\omega_{r'}^{j \to i}$, $b_{}^{j \to i}$ are the parameters in corelative regression.
The parameters are learned with the sum of the squared reconstruction error $\mathcal{L}_{corr}=ssr_{}^{j \to i}$.


\textbf{Granger Causality Relation Estimation.}
We estimate the causality relation by analyzing the two influences with the Granger causality test.
The Granger causality test explains the distribution of the causality relation over the difference between two influences.
The actor features are supposed to obey the Gaussian distribution, which can imply the influence, i.e. sum of squares residual (SSR) of the reconstructed feature, obeys the $ \chi^2 $ distribution.
Granger causality test uses the test statistic $\ f_{}^{j \rightarrow i}$ to analyze two $ \chi^2 $ distributions as:
\begin{small}
\begin{equation}
	\begin{split}
	\begin{aligned}
		\ f_{j \rightarrow i} = \frac{\left(ssr_{}^{ j \rightarrow i}-ssr_{}^{i}\right) / m}{ssr_{}^{ i} /\left(n_{m}-v_{m}\right)} \\
	\end{aligned}
\end{split}
\end{equation}
\end{small}
The test statistic obeys the Fisher-Snedecor distribution, which has two degrees of freedom.
$ n_{m}=(T-m) D $ is the sample number. 
$ v_{m}=2m+1 $ is the degrees of freedom of the corelative regression.
The Fisher-Snedecor distribution $\psi_F(\cdot)$ can project the test statistic into the causality probability value as:
\begin{small}
\begin{equation}
	p_{j \to i}=\int_{0}^{f_{j \to i}} \psi_{F(m,n_{m}-v_{m})}(z) dz 
\end{equation}
\end{small}


Our graph focuses on learning multiple causality relations by considering multiple asynchronous temporal features.
Multiple asynchronous temporal features are learned by temporal shifting with different temporal delays in the correlative regression.
The causality relation with the largest probability indicates the estimated temporal delay $delay_{j \to i}^{*}$ and the estimated causality probability $p_{j \to i}$ between two actions as:
\begin{small}
\begin{equation}
	[delay_{j \to i}^{*}, p_{j \to i}^{*}] = argmax_{delay} \  p_{j \to i} 
\end{equation}
\end{small}

Our graph represents the causality relations $A^{Granger }=\{a_{j \rightarrow i}^{\text {Granger }} \}$.
Each causality relation between two actors is detected by comparing the causality probability with a causality threshold as:
\begin{small}
\begin{equation}
	a_{j \rightarrow i}^{\text {Granger }}= 
\begin{cases}
1, & p_{j \to i}^{*}>\tau \\ 
0, & \text { otherwise }\end{cases}
\end{equation}
\end{small}


\subsection{Causality Features Fusion Module}
The causality relation indicates the cause actor and the effect actor in the group activity.
The effect action can be enhanced by integrating it with the cause action.
The features of the cause actor are synchronized by temporally shifting to the effect actor with the estimated delay.
Due to we cannot observe the features before the input frames, the early part of the shifted cause action features cannot be directly estimated.
We fill this early part using the feature of the first frame.
{\color{black}{The temporal shift operation is formed as:}}
\begin{small}
\begin{equation}
	\begin{split}
x_{k}^{shift, j \to i} = 
\left\{\begin{aligned} 
&x^j_{1},\quad k-delay^*_{j\to i}<1\\
&x^j_{k-delay^*_{j\to i}},\quad otherwise\\
\end{aligned}
\right.
\end{split}
\end{equation}
\end{small}

We use channel-wise concatenation to integrate the effect action features with the shifted cause action features.
The impact of each part is considered by introducing the channel ratio parameter $d$.
The input cause action feature has the shape $\mathbb{R}^{D}$.
The cause action feature is projected with the parameter $w_{i}^{d}$ in an FC layer to the shape $\mathbb{R}^{D/d}$.
The effect action feature is projected with the parameter $w_{j \to i}^{d}$ to the shape $\mathbb{R}^{D-D/d}$.
{\color{black}{We concatenate them and analyze the impact of the cause action feature by adjusting the parameter in two projection layers as follows:}}
\begin{small}
\begin{equation}
	\begin{split}
		&\ x_{k}^{con,j \to i} = concat(w_{i}^{d}  x^i_{k},w_{j \to i}^{d}  x_{k}^{shift,j \to i}) \\
\end{split}
\end{equation}
\end{small}


In the group activity, the centric actor may have multiple causality relations to different cause actors.
Each causality relation indicates a cause action to learn effect action features separately.
{\color{black}{We average the features learned with multiple causality relations as:}}
\begin{small}
\begin{equation}
	x_{k}^{syn,i}=\frac{1}{\sum_{j}a_{j \to i}^{Granger}}\sum_{j}x_{k}^{con,j \to i} 
\end{equation}
\end{small}
Our graph uses the node representation of all actors as $V^{cau} = \{ X^{syn}_i\}$, where each actor has the feature $X^{syn}_i =\{x^{syn,i}_{k}\}$.




\subsection{Causality Relation Graph Inference Module}
Our graph represents nodes with causality fusion features and represents the edges with causality relations.
The nodes describe effect action with asynchronous fused features by detecting the temporal delay, which is complementary to the synchronous features at the same timestamp learned with the temporal transformer.
The edges using causality relations can enhance the relevant relations in the spatial transformer.
To detail the causality relation with the actor's appearance and position, we embed the causality relation with the appearance relation and distance relation.

Following \cite{arg}, $h$ graphs are used to learn appearance relations, which is estimated by the dot product of two actor's features as:
$a^{app}_{i,j,h} = \frac { (w^i_{h} x^i)^T  (w^j_{h} x^j)}{\sqrt{D}}$.
The distance relation considers $s$ masks with different distance ratios $\lambda_s$ to limit the relation based on the image width $width$.
When the distance between actor $i$ and actor $j$ is smaller than the distance ratio $dist_{i,j} \leq \lambda_s  width$, the distance relation is $a^{dist}_{i,j,s}=1$.
Otherwise, the distance relation is 0.
Our graph embeds the causality relation with the appearance relation and distance relation to learn the causality edges $E_{h,s}^{cau}=\{ e_{h,s}^{j \to i}\}$ as:
\begin{small}
\begin{equation}
e_{h,s}^{j \to i} = 
\frac{a_{j \to i}^{Granger}  a_{i,j,h}^{app} a_{i,j,s}^{dist} } 
{\sum_{j}a_{j \to i}^{Granger}   a_{i,j,h}^{app} a_{i,j,s}^{dist}}
\end{equation}
\end{small}

Our graph learns the output features of nodes with multiple edges as:
\begin{small}
\begin{equation}
		X' = \sum_{h,s} ReLU( E_{h,s}^{cau} V^{cau} W_{h,s}^{graph})\\
\end{equation}
\end{small}
The output features of our actor-centric causality graph consider the asynchronous temporal causality relation.
The graph in the base model considers the synchronous temporal relation with the transformer-based network.
Two graphs are added together to enhance the actor representation.

\section{Experiments}

\subsection{Datasets and Implementation Details}

\textbf{Datasets.}
We conduct experiments on two widely-adopted group activity datasets which contain tracking annotations and bounding boxes, including the Volleyball dataset \cite{hdtm}and the Collective Activity dataset \cite{strap}. 
The metric employs the Multi-class Classification Accuracy \cite{arg}.

The Volleyball dataset\cite{hdtm} contains 55 video recordings of volleyball games and is split into 3493 training clips and 1337 testing clips. 
The center frame of each clip is annotated with bounding box coordinates for all actors and their action labels (i.e. blocking, digging, falling, jumping, moving, setting, spiking, standing, and waiting). 
Each clip is annotated with one group activity label out of eight labels (i.e. right set, right spike, right pass, right winpoint, left set, left spike, left pass, and left winpoint). 

The Collective Activity dataset\cite{strap} contains 44 clips. 
We follow the train set and test set in \cite{arg}. 
The center frame of every ten frames is annotated with bounding box coordinates of all actors and their action labels (i.e. NA, crossing, waiting, queueing, walking, and talking). 
Every ten frames are given one group activity label out of five (i.e. crossing, waiting, queueing, walking, and talking). 

\textbf{Implemenation Details.}
For feature extraction, we adopt the ImageNet pre-trained Inception-v3 \cite{inception-v3} as the backbone. 
The RoIAlign\cite{arg} is applied to extract the actor feature with each bounding box. 
The feature is embedded into $D=1024$ channels with an FC layer.
Besides the Inception-v3 features, we consider the pose feature with the ImageNet pre-trained HRNet-48 as backbone \cite{hrnet}.
The pose feature is extracted with each bounding box and concatenates with the Inception-v3 feature to enhance the actor feature.
The concatenated feature is embedded into $D$ channels with an FC layer.
Following \cite{gf}, the base model takes the group token number $K=8$ in the group encoder.
The spatial encoder and temporal encoder use 8 attention heads.
In the Actor Centric Causality Graph, we set the window size $m=4$, the set of temporal delay $delay = \{0,1,2\}$, the causality threshold $\tau=0.9$, the channel ratio parameter $d=6$.
For causality relation graph inference, we set the appearance graph number $k=16$, and the set of distance ratios $\lambda_s = \{0.1,0.2,0.3,0.4\}$.

We train our model in three stages. 
First, the parameters in the base model are trained with the base model loss.
Second, the parameters in the Asynchronous Granger Causality Detection Module, including the self-regression and correlative regression, are trained in each clip.
Third, the framework combines our causality relation graph and the base graph.

In the first and third training stage, the stochastic gradient descent with ADAM is adopted as an optimizer with fixed hyper-parameters to $\beta_1$ = 0.9, $\beta_2$ = 0.999, $epsilon$ =$10^{-8}$.
For the Volleyball dataset, we train the network in 150 epochs in each stage with a mini-batch size of 32 and a dropout ratio of 0.3 \cite{arg}. 
For the Collective Activity dataset, we train the network in 80 epochs in each stage with a mini-batch size of 16 and a dropout ratio of 0.5 \cite{arg}. 
For both datasets, the initial learning rate is 0.00001 and decreases by 0.1 after every 40 iterations.

{\color{black}{In model prediction, we use each test video clip to update the parameters of the self-regression and correlative regression, and fix the parameters of the rest modules.}}

\begin{table}[t]
% \setlength\tabcolsep{3pt}
\small
    \centering
	\caption{Comparisons with the state-of-the-art methods on Volleyball dataset and Collective Activity dataset.}
\scalebox{0.75}{
    \begin{tabular}{lccccc}
      \hline
	\multirow{2}{*}{Method} & \multirow{2}{*}{Flow}  & \multirow{2}{*}{Backbone} & \multicolumn{2}{c}{Volleyball} & \multicolumn{1}{c}{Collective Activity} \\
& & & Group & Individual & Group \\ 
      \hline
      HDTM\cite{hdtm}     &      & AlexNet      & 81.9     & -     & 81.5     \\
      CERN\cite{cern}     &      & VGG16     & 83.3     & -     & 87.2   \\
	CCGLSTM \cite{ccglstm} &      & AlexNet      & 89.3     & -     & 93.0     \\
      HRN\cite{hrn}     &      & VGG19     & 89.5     & -     & -     \\
      SSU\cite{ssu}     &      & Inception-v3     & 90.6     & 81.8     & -     \\
      PRL\cite{prl}     & & VGG16         & 91.4     & -        & 93.8     \\
      PDAR\cite{pdar}     &      & Inception-v3     & 92.2     & -     & 90.3     \\
      ARG\cite{arg}     &      & Inception-v3     & 92.5     & 82.8     & 91.0     \\
      AT\cite{at}     & \checkmark     & I3D     & 93.0     & 83.7     & 92.8     \\
      GLIL\cite{glil}     &      & Inception-v3       & 93.0     & -     & 94.9   \\
      VC\cite{vc}     &     & Inception-v3     & 93.3     & -     & 95.1    \\
      CRM\cite{crm}     & \checkmark     & I3D     & 93.0     & -     & 85.8     \\
      DIN\cite{din}     &      & VGG16     & 93.6     & -     & 95.9    \\
      GF\cite{gf}     &      & Inception-v3     & 94.1     & 83.7     & 93.6  \\
	 GRAIN \cite{grain}	&  & VGG16 & 94.5 & - & 95.2 \\
      SAACRF\cite{saacrf}    & \checkmark     & I3D+HRNet     & 96.4     & 85.5     & 96.0     \\
\hline
	Base model &      & Inception-v3     &   93.6   &  83.8     & 92.6        \\
     {\color{black}{Base+ACCG}}     &     & Inception-v3     &   95.0   &  85.6     &  94.5      \\
	{\color{black}{Base+ACCG}}     &     & VGG-16     &   95.5   &  85.8     &  95.0      \\
	Base+ACCG     & \checkmark      & I3D+HRNet     & \textbf{96.7}     & \textbf{86.4}     & \textbf{96.3}     \\
      \hline
    \end{tabular}}
\label{state} 
\end{table}

\begin{table}[t]
\small
	\centering	
	\caption{The effect of the temporal delay setting.}
\scalebox{0.75}{
    \begin{tabular}{ccccc}
    \hline
   \multirow{2}{*}{Delay}               & \multicolumn{2}{c}{Group} & \multicolumn{2}{c}{Individual} \\
                       & wo shift      & adaptive      & wo shift      & adaptive      \\
    \hline
    \{0\}       & 93.6             & -             	& 84.5             & -             \\
    \{0,1\}     & 94.4             & 94.7             & 85.1             & 85.3             \\
   \{0,1,2\}   & \textbf{94.5}             & \textbf{95.0}             & \textbf{85.2}             & \textbf{85.6}             \\
    \{0,1,2,3\} & \textbf{94.5}             & \textbf{95.0}             & \textbf{85.2}             & \textbf{85.6}            \\
    \hline
    \end{tabular}}
\label{delay}
\end{table}

\subsection{Comparison with the State-of-the-Art}

\textbf{Volleyball Dataset.}
Table \ref{state} shows the performance results on the Volleyball dataset. 
The base model is reported without considering the influences of actors.
Our model embeds the ACCG to analyze the influences of two actors with asynchronous features, which can detect asynchronous causality relations.
The asynchronous causality relation is complementary information to the synchronous temporal relation.
The model adds ACCG features and Base features to boost the performance.
The Base+ACCG outperforms the model with spatial-temporal relation learned in GF \cite{gf}.
Using the Flow, I3D, and HRNet\cite{hrnet} features, the Base+ACCG outperforms the SAACRF \cite{saacrf}.


\textbf{Collective Activity Dataset.}
Table \ref{state} shows the performance results on the Collective Activity dataset.
The Base+ACCG model considers both the synchronous relation and the asynchronous causality relation learned by analyzing the influences of actors.
The Base+ACCG model with RGB feature outperforms the GF method \cite{gf}.

\subsection{Ablation Study}
To dispel any concerns that the improvement is simply from additional optical flow and pose information, we perform ablation studies using RGB features learned with the Inception-v3 backbone on the Volleyball dataset.




\begin{table}[t]
\small
	\centering
	\caption{The effect of channel ratio parameter. }
% 	\renewcommand\arraystretch{2}
\scalebox{0.75}{
	\begin{tabular}{ccc}
		\hline
		{\color{black}{Channel ratio parameter}}             & Group  & Individual  \\ 
		\hline
		without 	& 93.3 	  &	83.7\\
		d=2 		& 94.1    & 84.7    \\
		d=4        	& 94.6    & 85.3    \\
 		d=6     	& \textbf{95.0}    & \textbf{85.6}    \\ 
		d=8  		& 93.5    & 84.0    \\ 
		\hline
	\end{tabular}}
\label{channel}
\end{table}


\begin{table}[tbp]\centering
	\caption{The effect of graph number and distance mask number.} 
\scalebox{0.75}{
	\begin{tabular}{ccccccccc}
		\hline
		 \multirow{3}{*}{Appearance graph} &  \multicolumn{3}{c}{Group} &  \multicolumn{3}{c}{Individual} \\ 
&  \multicolumn{3}{c}{Distance mask} &  \multicolumn{3}{c}{Distance mask}\\
			  & 2   & 3  & 4     & 2 & 3  & 4 \\ \hline
	8 		& 93.5	& 94.5	& 94.8		& 84.1	& 85.1	& 85.6\\
	16		& \textbf{93.7}	& \textbf{94.7}	& \textbf{95.0}		& \textbf{84.1}	& \textbf{85.1}	& \textbf{85.6}\\
	32 		& 93.6	& 94.6	& 94.9		& 84.0	& 85.0	& 85.5\\
		\hline
	\end{tabular}}
\label{graph}
\end{table}


\begin{table}[t]
% \setlength\tabcolsep{3pt}
\small
    \centering
	\caption{The effect of graph attention.}
\scalebox{0.75}{
    \begin{tabular}{lcccc}
      \hline
	\multirow{2}{*}{Method}    & \multicolumn{2}{c}{Volleyball}  \\
 & Group & Individual \\ 
      \hline
		ARG \cite{arg}             & 92.5     & 82.8          \\
		ACCG wo attention              & 93.3     & 83.1        \\
     	     ACCG          & 93.6     & 83.7        \\
          ARG+ACCG          & \textbf{93.9}     & \textbf{84.0}        \\
\hline
		Base model         &   93.6 & 83.8            \\
		Base+ACCG wo attention          &   94.3 & 84.7           \\
	  	Base+ACCG         &   \textbf{95.0} & \textbf{85.6}            \\
      \hline
    \end{tabular}}
\label{attention} 
\end{table}

\begin{table}[t]
\small
	\centering
	\caption{Comparison with other graph relation learning. GD, SE, and TE denote the Group decoder, Spatial encoder, and Temporal encoder in the base model. $V^{cau}$, $E^{cau}$ denote the node representation and edge relation in the causality relation graph.}
% 	\renewcommand\arraystretch{2}
\scalebox{0.75}{
	\begin{tabular}{cccc | cc | cc}
		\hline
 		\multicolumn{4}{c|}{Base model} & \multicolumn{2}{c|}{Causality graph} &\multirow{2}{*} {Group}  &\multirow{2}{*} {Individual}\\
		Inception-v3 & GD   & SE  & TE & $V^{cau}$ & $E^{cau}$ \\ 
		\hline
		\checkmark  & & & &  & & 89.8    & 80.9    \\
		\checkmark & \checkmark & &   & & & 91.0 & 82.1\\
		\checkmark & \checkmark & \checkmark &   & & & 91.8 & 82.2\\
		\checkmark & \checkmark & \checkmark &  \checkmark & & & 93.6 & 83.8\\
\hline
		\checkmark &  & &   & \checkmark & & 91.9    & 82.5    \\
          \checkmark & &  &    & & \checkmark & 92.9    & 83.3    \\ 
          \checkmark &   &  &   & \checkmark & \checkmark & 93.6    & 83.7    \\ 
		\checkmark & \checkmark & \checkmark &  \checkmark & \checkmark & \checkmark & \textbf{95.0}    & \textbf{85.6}    \\ 
		\hline
	\end{tabular}}
\label{exist}
\end{table}

\textbf{The effect of the temporal delay setting.}
{\color{black}{Table \ref{delay}}} analyzes the model with different temporal delays.
We consider multiple asynchronous temporal features by setting multiple temporal delays.
In {\color{black}{Table \ref{delay}}}, the model without temporal shifting (wo shift) learns the causality feature with the original correlative features.
In {\color{black}{Table \ref{delay}}}, the model with adaptive temporal shifting (adaptive) learns the causality feature with the synchronized correlative features.
The model with adaptive temporal shifting outperforms the model without temporal shifting.

\textbf{The effect of channel ratio parameter.}
{\color{black}{Table \ref{channel}}} analyzes the model with different channel ratio parameters, which are used to adjust the impact of the cause action and the effect action.
The model without considering the cause action features gets the worst performance.
We introduce the channel ratio parameter $d$ to enhance the effect action features with cause action features.
We increase the channel ratio parameter to enlarge the impact of effect action and get the best performance at $d=6$.

\textbf{The effect of appearance graph number and distance mask number.}
{\color{black}{Table \ref{graph}}} analyzes the model with multiple appearance graphs in the graph relation inference.
The model with 16 appearance graphs gets enough appearance relations.
For distance relation learning, we adopt 4 distance masks.
The model with 4 distance mask and 16 appearance graphs get the best performance.

\textbf{The effect of graph attention.}
Table \ref{attention} provides the ACCG with/without graph attention. 
ACCG model considers ARG [32] relation, which uses graph attention to learn the appearance relation. 
ACCG uses asynchronous temporal relations to select crucial ARG relations. 
We provide ACCG without (wo) attention by removing the ARG relation from ACCG.
ACCG wo attention uses a set of temporal delays to capture asynchronous temporal relations, and outperforms the ARG model. 
We provide a two-branch model ARG+ACCG to adopt the ARG as the base model. 
ARG+ACCG integrates two branch relations in a complementary way, and outperforms ARG and ACCG.


\textbf{Comparision with other graph relation learning.}
{\color{black}{Table \ref{exist}}} analyzes the contribution of each component.
The base model uses GD, SE, and TE to learn the actor relation and improve performance.
In the causality graph, our model enhances the node features by fusing two actors' features by synchronizing with the temporal delay detected in the causality relation.
The causality fused features help to increase the performance.
Our model learns asynchronous causality relations to indicate relevant relations in the group activity for better relation learning.
Without considering the synchronous relation learned in the base model, our causality graph with asynchronous causality relation outperform the base model.
The asynchronous causality relation is complementary to the transformer-based relation.
We integrate the two relations to get the best performance.
The Base model has 63.6 MParams and 408.5 GFLOPs.
The Base+ACCG has 89.8 MParams and 414.8 GFLOPs.



\begin{figure}[t] 
\centering  
\includegraphics[width=0.5\textwidth]{figs/vis.png} 
\caption{Visualization of the causality relation in the group activity.
The changed actor labels are visualized.}
\label{vis} 
\end{figure}

\subsection{Visualization}

{\color{black}{Figure \ref{vis} shows two examples of the Volleyball dataset.}}
In (a), The base model considers the moving actor (No.6) as passing the volleyball and mispredicts the group activity as a “Right pass”.
{\color{black}{Our actor-centric causality graph estimates the correlative influence of two actors (No.7 and No.8), which is larger than the self influence (No.8).
Our graph analyzes these two influences to indicate the causality relation from No.7 to No.8.}}
Our graph also detects the causality relation from No.9 to No.8.
The two causality relations enhance the action of No.8 to correct its prediction as the setting.
Besides, our causality relation explains that No.2 is waiting for the volleyball which passed from No.8.
In (b), our graph detects two causality relations, which explain that No.11 is Digging the ball passed from No.3 and No.11 is adjusting the action to pass the ball to No.7.
Besides, our causality relation explains that No.10 is moving to spike the volleyball which is set by No.11.


\section{Conclusion}
In this work, we propose an actor-centric causality graph, which focuses on analyzing the influence of two actors for asynchronous causality relation detection.
We learn the causality fused feature by integrating the effect action features with the synchronized cause action features.
The learned asynchronous relation is complementary to the synchronous relation learned in the transformer-based model.
We integrate two relation models, which can outperform the state-of-the-art methods.

\section{Acknowledgments}
This work is supported in part by the National Natural Science Foundation of China under Grant 62272144.

% %%%%%%%%% REFERENCES
 {\small
 \bibliographystyle{ieee_fullname}
%\bibliographystyle{unsrt}
 \bibliography{refer}
 }


\end{document}
