\begin{abstract}
 Multimodal depression detection through internet-based data such as social media platforms has been an important problem in the research community, aiming to predict human mental states for ensuring wellbeing of the society. Recently, attention-based networks have gained significant popularity for depression detection. However, existing multimodal methods primarily rely on images and text assuming no correlation between temporal aspects such as relative time of different posts or tweets, which is a crucial factor in deriving depression related behavior patterns. Moreover, they lack model interpretability resulting in limited understanding of how different features are contributing to the model's final prediction. In this paper, we propose Multi-Explainable TemporalNet (METN), a Temporal Convolution Network (TCN) based multi-modal transformer network with relative timestamp embeddings. We leverage pretrained foundation models for text and image embeddings and attention maps for model interpretability. We perform extensive experiments and ablation studies to validate the performance of METN for user-level depression detection task. Our model shows state-of-the-art results on various benchmarks, such as \textbf{0.945} F1 score on multimodal Twitter dataset, and \textbf{0.913} F1 score on multimodal Reddit dataset. We further demonstrate that our model enhances the accuracy of identifying depression in individuals who publicly post messages on social media platforms with enhanced interpretable compatibility. Code and models are available at \href{https://github.com/anas-zafar/MTEN}{Github}.
 \footnote{* Corresponding author}

 %\textbf{\textit{Index terms -----} Depression Detection, Transformer, Temporal Convolution Networks (TCN), Explainable, Social Media, Multimodal, Computer Vision, Deep Learning}
\end{abstract}
\begin{figure*}[!htb]
    \centering
    \begin{subfigure}{\textwidth} % Specify width here; \textwidth makes it span the whole width of the text area
        \centering
        \includegraphics[width=18cm, height=5cm]{author-kit-CVPR2024-v2/Architecture-latest.pdf}
        % \caption*{} % Uncomment and use if you need a sub-caption without a label
    \end{subfigure}
    \caption{Overview of the MTEN architecture for user-level depression detection task. A window size of \textit{n} posts are sampled from a user’s profile containing images, texts and time stamps. The images and text are then encoded using pretrained encoders which are then fused with temporal embeddings obtained from the TCN encoder. These are further refined by a bilinear pooling layer and fed into a cross modal encoder. Self-attention and cross attention is then applied on these embeddings which are then later fed into a classification head for depression detection.}
    \label{fig:fig1}
\end{figure*}