\section{Method}

\begin{table*}[htb]
\centering
\footnotesize
\begin{tabular}{l|c|cccc}
\toprule
\textbf{Method} &\textbf{ Modality }&\textbf{F1}&\textbf{Prec}&\textbf{Recall}&\textbf{Acc}\\ \midrule
T-LSTM \cite{baytas2017patient} & T & 0.848$\pm$8e-3 & 0.896$\pm$2e-2 & 0.804$\pm$1e-2 & 0.855$\pm$5e-3 \\
EmoBERTa Transformer & T & 0.864$\pm$1e-2 & 0.843$\pm$1e-2 & 0.887$\pm$3e-2 & 0.861$\pm$1e-2 \\
LSTM + RL \cite{gui2019depression} & T & 0.871 & 0.872 & 0.870 & 0.870 \\
CNN + RL \cite{gui2019depression} & T & 0.871 & 0.871 & 0.871 & 0.871 \\ \midrule
MTAL \cite{an2020multimodal} & T+I & 0.842 & 0.842 & 0.842 & 0.842 \\
GRU + VGG-Net + COMMA \cite{gui2019cooperative} & T+I & 0.900 & 0.900 & 0.901 & 0.900 \\
MTAN \cite{cheng2022multimodal} & T+I & 0.908 & 0.885 & 0.931 & - \\ 
Vanilla Transformer \cite{bucur2023s} & T+I & 0.886$\pm$1e-2 & 0.868$\pm$2e-2 & 0.905$\pm$2e-2 & 0.883$\pm$5e-3 \\
SetTransformer \cite{bucur2023s} & T+I & 0.927$\pm$8e-3 & 0.921$\pm$1e-2 & 0.934$\pm$2e-2 & 0.926$\pm$8e-3 \\
Time2VecTransformer \cite{bucur2023s} & T+I & 0.931$\pm$4e-3† & 0.931$\pm$2e-2† & 0.931$\pm$1e-2† & 0.931$\pm$4e-3† \\ 
\midrule
\textbf{MTEN (Ours)} & \textbf{T+I}  & \textbf{0.945$\pm$1e-2} & \textbf{0.945$\pm$2e-2 } & \textbf{0.945$\pm$2e-2}& \textbf{0.945$\pm$5e-3}\\ 
\bottomrule
\end{tabular}
\caption{Quantitative results of multimodal depression detection on twitter dataset\cite{gui2019cooperative}}
\label{tab:tab2}
\end{table*}

\begin{table*}[!htb]
\centering
\begin{tabular}{l|c|cccc}
\hline
\textbf{Method} & \textbf{Modality} & \textbf{F1} & \textbf{Prec.} & \textbf{Recall} & \textbf{Acc} \\
\hline
T-LSTM \cite{baytas2017patient} & T & $0.831\pm0.01$ & $0.825\pm0.008$ & $0.837\pm0.01$ & $0.872\pm$7e-3 \\
EmoBERTa Transformer & T & $0.843\pm0.006$ & $0.828\pm0.003$ & $0.858\pm0.01$ & $0.879\pm$4e-3 \\
Uban et al. \cite{uban2022explainability} & T+I & - & - & - & 0.663 \\
Vanilla Transformer \cite{bucur2023s} & T+I & $0.837\pm0.008$ & $0.827\pm0.01$ & $0.848\pm0.01$ & $0.876\pm$6e-3 \\
Set Transformer \cite{bucur2023s} & T+I & \textbf{$0.902\pm0.007$}† & \textbf{$0.878\pm0.006$}† & \textbf{$\textbf{0.928}\pm\textbf{0.01}$}† & \textbf{$0.924\pm5e-3$}† \\
Time2Vec Transformer \cite{bucur2023s} & T+I & $0.869\pm0.007$ & $0.869\pm0.007$ & $0.869\pm0.008$ &$0.901\pm5e-3$\\
% \hline
\midrule
\textbf{MTEN (Ours)} & \textbf{T+I}  & \textbf{0.913$\pm$0.05} & \textbf{0.913$\pm$0.05} & 0.913$\pm$0.05& \textbf{0.926}$\pm$\textbf{0.002}\\
\bottomrule
\end{tabular}
\caption{Quantitative results of multimodal depression detection on the Reddit dataset \cite{Uban2022}.}
\label{tab:tab3}
\end{table*}
\subsection{Framework}
The proposed MTEN is a TCN based multi-modal transformer incorporating temporal cues such as posting time stamps along with image and textual features along with an interpretable module for better understanding of different features, as shown in Figure~\ref{fig:fig1}. Many existing approaches struggle with long-range temporal information and modality fusion. In contrast MTEN, effectively captures long-range dependencies and temporal information over long sequences. It efficiently fuses different modalities while preserving rich features leading to more accurate predictions. It consists of the following properties; (i) the ability to capture long-range temporal dependencies (ii) self and cross attention mechanism for preserving features and attention maps for interpretability (iii) effective modality fusion. 
\subsubsection{Model Pipeline}

There is a User $U$, who has multiple user posts $U_{p}$ such that each post has a posting time \textit{J}, an image \textit{I} and text \textit{T}. Using a post sequence $P_{x}$ is defined as $S$ sampled posts from user $U_p:P_x = \left\{ (T^{y}, I^{y}, \Delta^{y}) \sim U_{p} \mid y \in (1 \ldots S) \right\}$ where $y$ is a specific post within the user's post-sequence and $\Delta$ is the posting time. Different window sizes of user posts are sampled from the user's profile which is then used as an input batch for training. The images and text are encoded using pretrained foundation models; CLIP\cite{radford2021learning} for images and EmoBERTa\cite{kim2021emoberta} for textual data due to their ability to accurately capture these features. Contrastive Language-Image Pre-training (CLIP) is a a vision transformer that is trained to match images with corresponding textual descriptions, enabling it to perform well on various visual tasks without task-specific training. EmoBERTa utilizes both textual and emotional data from users' posts. Instead of using same encoders, we use two different encoders, as foundation models pretrained on ImageNet tend to output general purpose embeddings. Given the diversified nature of our dataset these general purpose embeddings are not suitable for our usecase. The posting time is encoded through a TCN encoder which is able to capture long-range dependencies and handles very long input sequences efficiently as it computes all outputs in parallel during training. The TCN encoder contains causal convolution layers which preserves the temporal information of the user’s post is able to capture long range dependences due to dilated convolution present in its architecture. 

We use Self-DIstillation with NO Labels (DINO)\cite{caron2021emerging} as regularizer for CLIP generated embeddings for enriched feature representations and making it more robust to noise and variations in the input data. DINO is a self-supervised vision transformer that directly predicts the output of a teacher network using cross entropy loss. During this process CLIP embeddings are fed into DINO, which refines these embedding by making it more consistent with DINO trained network’s output. The image and the text embeddings are then projected to a fixed size through a learnable linear feed-forward layer. The attention maps obtained from the text and image encoder are used for model interpretability and explainability, of how different features are contributing the output as show in Figure~\ref{fig:fig2}. To make it time-aware, we fuse these image and text embeddings with temporal embeddings generated by the TCN encoder followed by bi-linear pooling layer to capture all pairwise interactions between features from these modalities, hence improving model's robustness to noise and variations. Although, EmoBERTa and CLIP have distinct latent spaces, in our approach we leverage ViLBERT (Vision-and-Language BERT)\cite{lu2019vilbert}, a cross modal encoder for the alignment between these two modality embeddings. ViLBERT supports cross-modal interactions through its co-attention transformer layers. It allows joint representation learning by bridging the gap between the two generated embeddings from CLIP and EmoBERTa. Through this it ensures that the features from both the modalities are effectively fused and are aligned in the latent space hence obtaining richer representation for joint learning representations.

We also use multimodal dropout in which we randomly drop modalities during training to make our model more robust to missing data. The embeddings are then processed through cross and self-attention across different user posts, after which they are finally fed into a classification head for depression detection. We use binary cross entropy loss for training this model.
\begin{table}[!htb]
\centering
\begin{tabular}{cccc}
\toprule
\textbf{Dataset} & \textbf{Class} &\textbf{T}  & \textbf{(T+I)}\\
\midrule
Reddit&Depr & 6,6M & 46.9K  \\
&Non-Depr & 8,1M & 73.4K  \\
\midrule
Twitter&Depr &213K& 19.3K  \\
&Non-Depr & 828K & 50.6K  \\
\bottomrule
\end{tabular}
\caption{Data distribution of different modalities using benchmark Reddit and Twitter dataset, T represents number of posts having only text and T+I represents number of posts having both images and text}
\label{tab:dataset}
\end{table}
\begin{table*}[!htb]
\footnotesize
\centering
\begin{tabular}{l|c|c}
\toprule
 & \multicolumn{1}{c|}{\textbf{Twitter}} & \multicolumn{1}{c}{\textbf{Reddit}} \\
 \midrule
\textbf{Text+Image Enc}&\textbf{F1}&\textbf{F1}\\
\midrule
EmoBERTa+CLIP & 0.930 & 0.891 \\
EmoBERTa+DINO & 0.916 & 0.873 \\
EmoBERTa+CLIP+LXMERT & 0.934   & 0.897  \\
EmoBERTa+DINO+LXMERT & 0.932 & 0.894\\
EmoBERTa+CLIP+LXMERT (Dino Regularizer) & 0.936& 0.899\\
EmoBERTa+CLIP+ViLBERT & 0.943 & 0.911\\
EmoBERTa+DINO+ViLBERT & 0.941 & 0.908\\
\textbf{EmoBERTa+CLIP+ViLBERT (Dino Regularizer)} & \textbf{0.945}&\textbf{0.913}\\
\bottomrule
\end{tabular}
\caption{Ablation study results on MTEN using different image encoders and cross modal encoders on Twitter and Reddit dataset}
\label{tab:tab4}
\end{table*}
\begin{table}[ht]
\footnotesize
\centering
\begin{tabular}{c|c|c}
\hline
\textbf{Dataset} & \textbf{Window Size} & \textbf{F1-Score} \\
\hline
                         & 16 & 0.933 \\
                         & 32 & 0.938 \\
                 Twitter & 64 & 0.941 \\
                         & 128 & 0.943 \\
                         & 256 & 0.945 \\
                         & 512 & 0.942 \\
\hline
                         & 16 & 0.895 \\
                         & 32 & 0.905 \\
                 Reddit  & 64 & 0.910 \\
                         & 128 & 0.913 \\
                         & 256 & 0.912  \\
                         & 512 & 0.910 \\
\hline
\end{tabular}
\caption{Ablation study results on different window sizes on both twitter and reddit dataset}
\label{tab:tab5}
\end{table}
For model interpretabilty we leverage attention maps from text and image encoder as shown in Figure~\ref{fig:fig2}. The attention maps demonstrate how different features are contributing to the model prediction.This figure shows METN assigns higher weights to depressed tweets compared to less relevant tweets highlighting which tweets are more responsible to the final model output.

\section{Experiments and Results}
\subsection{Dataset \& Evaluation Metrics}
We trained MTEN using benchmark multimodal twitter dataset\cite{Gui_Zhu_Zhang_Peng_Zhou_Ding_Chen_2019}and multimodal Reddit dataset\cite{Uban2022} for user-level depression detection. The multimodal twitter dataset consists of 1,402 depressed users and 1,402 control users. Similarly, the reddit dataset has 1,419 depressed users and 2,344 control users. Reddit has a more detailed textual information containing upto 40,000 characters, whereas twitter dataset only contains 280 characters. Please refer to Table \ref{tab:dataset} for detailed data distribution for the two multimodal datasets. For training, we followed the same setting as Gui~\textit{et al.}\cite{gui2019cooperative} and Bucur~\textit{et al.}\cite{bucur2023s}. To evaluate the results, we calculate the Precision, Recall, F1-score and accuracy metrics.

\subsection{Implementation Details}
We trained the model using Adam\cite{kingma2014adam} optimizer with (γ1=0.9, γ2=0.9, weight decay 0) for 10 epochs. The initial learning rate is set to 1e−4, which gradually reduced to 1e−6 with the cosine annealing scheduler\cite{loshchilov2016sgdr}. The model consists of consists of four cross-encoder layers, each with eight heads, and an embedding size of 128. For evaluation purposes 10 samples per user is used for each classification label. We use the PyTorch framework to implement our architecture, and trained our model using NVIDIA DGX server with a NVIDIA A-100 GPU, having 40-GB RAM.

\subsection{Results and Comparison With Other Methods}
We evaluate our model on both multi-modal benchmark twitter and reddit dataset. Quantitative results are shown in Table~\ref{tab:tab2} and Table~\ref{tab:tab3} using precision, recall, F1 score and accuracy metrics. We compare our model against state-of-the-art multimodal and text-based baselines. MTEN was able to achieve state-of-the-art results on both the multimodal datasets with an f1 score of 0.945 F1 score on multimodal twitter dataset, and 0.913 F1 score multimodal reddit dataset respectively. Our model outperforms all the time-enriched models Time2VecTransformer\cite{bucur2023s}, SetTransformer\cite{bucur2023s}, T-LSTM\cite{baytas2017patient} and MTAN\cite{cheng2022multimodal} on both the datasets demonstrating that our model is able to accurately capture long-range temporal information. We also obtain attention maps for each modality for model interpretability demonstrating how different features are contributing to the models predictions and can seen in Figure~\ref{fig:fig2}.


\subsection{Ablation Study}
We performed an in depth ablation study to explain the role of
different components in our proposed MTEN architecture. We evaluated our model on both LXMERT\cite{tan2019lxmert} and  ViLBERT (Vision-
and-Language BERT\cite{lu2019vilbert} for the cross modal encoder as shown in Table \ref{tab:tab4}. MTEN(ViLBERT) can be observed to achieve a performance boost (0.94 F1 score) when
as compared to LXMERT for user-level depression detection. Similarly, we also compare DINO vs CLIP for image encoder and we can observe an improvement in the performance when we regularize CLIP with DINO embeddings.
\par To decide the best window size we also carried an ablation study on different window sizes $W=\{16,32,64,128,256,512\}$ as show in Table \ref{tab:tab5} using CLIP, EmoBERTa and ViLBERT for text and image embeddings. For twitter dataset, window size 256 is able to the highest accuracy of 0.945 F1 score. For reddit dataset, window size 128 is best suitable accurate predictions.