\section{Introduction}
Depression is one of the most common and crucial psychiatric disorder in today’s community. According to World Health Organization (WHO)\cite{huang2019prevalence}, the severity of depression, known as Major Depressive Disorder (MDD), hinders daily activities for over 280 million people globally, leading to suicides\cite{maurer2012screening}, self-harm and mass shootings\cite{metzl2015mental}, causing the loss of innocent lives. The symptoms of depression\cite{maurer2012screening} can vary drastically ranging from low self-esteem to extreme mood swings and cynical behavior. Due to its subjective nature, depression detection is a challenging problem in modern day community. Approximately 33\% of patients with depression are not identified during clinical diagnostic procedures, and less than 40\% receive appropriate treatment\cite{egede2007failure}. Therefore, a range of methods have been developed using assistive technologies for early diagnosis for depression detection\cite{reynolds2012early}.
Social media platforms such as twitter\cite{culotta2015predicting} and reddit\cite{jamnik2019use} have been very popular in today’s world, where users can share their thoughts and mental issues freely. Reddit contains different threads where individuals can discuss their problems with complete anonymity. Many methods\cite{yates2017depression,alhuzali2021predicting,kenton2019bert,agirrezabal2022kucst} leverage this social media user data for early depression detection using pretrained language models such as BERT\cite{devlin2018bert} and LSTM\cite{wang2015learning} based models, leading to significantly better results when compared with traditional machine learning approaches. Recent works\cite{gui2019cooperative,safa2022automatic,radford2021learning,xu2020inferring} such as multimodal depression techniques use both visual and text data for detecting depression, but fail to leverage temporal cues such as relative time between different tweets and post leading to degraded performance. Moreover, current state-of-the-art models\cite{zogan2021depressionnet,yadav2023novel,mcintyre2023treatment} lack model interpretability and explainability causing trust issues with the users, as they are not able to understand the driving features causing depression. 

To address these limitations, we propose Multi-Explainable TemporalNet (MTEN), a Temporal Convolution Network (TCN)\cite{lea2017temporal} based multimodal transformer with temporal embeddings for depression detection from social media datasets \cite{garg2023mental,Gui_Zhu_Zhang_Peng_Zhou_Ding_Chen_2019,Uban2022}. TCN is a framework that utilizes casual convolutions and dilations for sequential data addressing temporal data and large receptive fields. MTEN utilizes pretrained state-of-the-art models for generating text and image embeddings, and later fuses them with time stamp embeddings obtained through a TCN encoder. We leverage self-attention and cross-attention mechanisms that assigns weights to different parts of an image and sentences. The attention maps generated from these modules are used for model interpretability, in order to understand how different features in an image or text are contributing to the final model prediction. We used multimodal Twitter\cite{Gui_Zhu_Zhang_Peng_Zhou_Ding_Chen_2019} and multimodal Reddit\cite{Uban2022} datasets for training and evaluation purposes and our model achieved state-of-the-art results for user-level depression detection task.

Our main contributions in this work are:
\begin{itemize}
\item We propose a novel TCN based multi-modal transformer network (MTEN) which leverages temporal cues; relative time stamps, along with image and text embeddings for enhanced user-level depression detection and applied self and cross attention mechanism through which it dynamically adapts the fusion of weights for different modalities.
\item Within the MTEN Block, we introduce an interpretable module through attention maps for better understanding and explainability of how different features in an image or text are contributing to the model's final prediction.
\item We perform extensive experiments and ablation studies to validate the performance of METN. Our method achieves state-of-the-art results on multi-modal twitter dataset 0.945 F1 score and 0.913 F1 score on multi-modal reddit dataset.
\end{itemize}
The paper is organized as follow; Section 2 provides a comprehensive overview of related work in multi-modal depression detection task, and Section 3 presents our architecture in detail. Section 4 presents results and experiments, including an extensive ablation study, and qualitative and quantitative evaluations, comparing our approach with other multi-modal depression detection baselines. In Section V, we discuss some of the limitations of the proposed model, and Section VI summarizes our research findings and potential future works.



