



\documentclass[10pt,letterpaper]{article}
\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{makecell}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{color}
\usepackage{setspace}
\usepackage{pifont}
\usepackage{float}

% Include other packages here, before hyperref.

% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex.  (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[breaklinks=true,bookmarks=false]{hyperref}

\cvprfinalcopy % *** Uncomment this line for the final submission

\def\cvprPaperID{5641} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}

% Pages are numbered in submission mode, and unnumbered in camera-ready
\pagestyle{empty}
%\setcounter{page}{4321}
\begin{document}

%%%%%%%%% TITLE
\title{Self-supervised Domain-aware Generative Network for Generalized Zero-shot
Learning}

%\author{First Author\\
%Institution1\\
%Institution1 address\\
%{\tt\small firstauthor@i1.org}
%% For a paper whose authors are all at the same institution,
%% omit the following lines up until the closing ``}''.
%% Additional authors and addresses can be added with ``\and'',
%% just like the second author.
%% To save space, use either the email address or home page, not both
%\and
%Second Author\\
%Institution2\\
%First line of institution2 address\\
%{\tt\small secondauthor@i2.org}
%}

\author{Jiamin Wu\textsuperscript{$1$}, Tianzhu Zhang\textsuperscript{$1$}, Zhengjun Zha\textsuperscript{$1$}, Jiebo Luo\textsuperscript{$2$}, Yongdong Zhang\textsuperscript{$1$}, Feng Wu\textsuperscript{$1$}\\
\textsuperscript{$1$} University of Science and Technology of China 
\\
\textsuperscript{$2$} University of Rochester\\
{\tt\small jiaminwu@mail.ustc.edu.cn, \{tzzhang, zhazj, zhyd73, fengwu\}@ustc.edu.cn}\\
{\tt\small
 jluo@cs.rochester.edu}
}

\maketitle
\thispagestyle{empty}



\begin{figure*}[htp]
\centering
\includegraphics[width=1.0\textwidth]{graph/complete-framework.pdf}
\caption{
The architecture of our SDGN, which is composed of two modules. (1)
The cross-domain feature generating module is trained to synthesize features $\tilde{x}^s$ and $\tilde{x}^t$ for seen and unseen classes. It consists of one generator $G$, and two discriminators $D_s$ and $D_t$ for source and target domains respectively. The multi-labels are combined with target features to form pairs. They act as the input for the additional target domain discriminator $D^t$ to help preserve domain-label consistency.
(2) The self-supervised learning module
takes  $\tilde{x}^t$ and $\tilde{x}^s$ as an input,
and compares them with anchors $A$ to derive the soft multi-labels $\mathcal{M}^s$ and $\mathcal{M}^t$ for feature reconstructions. $\tilde{x}$ and its reconstructed feature $x^r$ in the same domain will be pulled closer while features from different domains will be pulled away. Constrained by $\mathcal{L}_{cross}^{\mathcal{T}}$, $\mathcal{L}_{cross}^{\mathcal{S}}$ and $D^t$, $\tilde{x}$ can evolve towards the highly-discriminative domain-aware representation.}
\label{a}
\end{figure*}

In the supplementary material, we first introduce the details of datasets, our model architectures and parameters on the five datasets,
respectively. Then, we show more t-SNE results on the generated features. Finally, we include our
source code for testing and give a brief introduction.



\section{Model Architectures and Parameters}





The overall architecture of the proposed model is shown in Figure~\ref{a}. To demonstrate the effectiveness of the proposed model, we conduct
experiments on five standard datasets including: CUB~\cite{wah2011caltech}, AwA1~\cite{lampert2013attribute}, AwA2~\cite{xian2017zero}, SUN~\cite{patterson2012sun} and FLO~\cite{nilsback2008automated}. Here, we present the exact model architectures used
for each experiment along with hyperparameters used to reproduce results.
%
\subsection{More Details on Datasets}
We give more detailed introduction of datasets used in the generalized zero-shot learning. 
CUB is a fine-grained dataset consisting of 200 species of birds, 
among which 150 classes are regarded as seen categories, 
and the other 50 classes are regarded as unseen. 
It is a medium scale dataset with total 11K images. 
AwA1 and AwA2 are coarse-grained datasets consisting of 50 kinds of animals, with 40 classes as seen categories and 10 classes as unseen categories. 
The size of AwA2 is larger than AwA1, i.e., 37K vs 30K. 
SUN is a fine-grained and medium-scale dataset 
with respect to both number of images and number of classes, 
i.e., SUN contains 14K images coming from 717 
types of scenes annotated with 102 attributes. 
We use 645 classes to compose source domain, 
and 72 classes to compose target domain. 
FLO is a fine-grained dataset with 8K images from 102 types of flowers, 
among which 82 classes used as seen categories and 20 classes are used as unseen categories. 
More details about datasets are shown in Table~\ref{Dataset}.
\begin{table}[]
\caption{The details of benchmark datasets used in GZSL.}\smallskip
\centering
\resizebox{0.5\columnwidth}{!}{
\smallskip\begin{tabular}{cccccc}
\toprule
Dataset & CUB & AwA1 & AwA2 & SUN & FLO \\
\midrule
Granularity & fine & coarse & coarse & fine & fine \\
Size & 11K & 30K & 37K & 14K& 8K  \\
$\mathcal{Y}$ & 200 & 50 & 50 & 717 & 102 \\
$\mathcal{Y}_\mathcal{S}$ & 150 & 40 & 40 & 645 & 82 \\
$\mathcal{Y}_\mathcal{U}$ & 50 & 10 & 10 & 72 & 20 \\
Att & 312 & 85 & 85 & 102 & 1024  \\
\bottomrule
\end{tabular}
}
\label{Dataset}

\end{table}



\subsection{On the CUB Dataset}
\textbf{Hyperparameters}:
\begin{itemize}
\item Batch size is 64.
\item Base learning rate is 0.0001.
\item The loss coefficient $\lambda_C$ of pre-trained classifier is 0.01.
\item The loss coefficient $\lambda_t$ of target domain classifier is 1.
\item The loss coefficient $\lambda_a$ of self-supervised learning module is 0.01.
\item Parameters are initialized from zero centered normal distribution with a standard deviation 0.02.
\item The synthesized feature number is 300.
\item The margin $\mu$ in the triplet mining loss is 0.3.
\item The number of epoch is 300.
\end{itemize}




\textbf{Generator} $G$:
\begin{itemize}
\item It has 2 fully connected layers. The first fully connected layer is followed by a LeakyReLU layer with a negative slope set as 0.2. the second fully connected layer is followed by a ReLU layer.
\item The input is the concatenation of a Gaussian noise vector and a class embedding vector.
\item The dimension of Gaussian noise is set as the same as the dimension of the class embedding.
\item We use the ADAM optimizer with $\beta=0.5$.
\end{itemize}


\textbf{Discriminator} $D_s$:
\begin{itemize}
\item It has 2 fully connected layers. The first fully connected layer is followed by a LeakyReLU layer with a negative slope set as 0.2.
\item The input is the concatenation of a source image feature and its class embedding.
\item We use the ADAM optimizer with $\beta=0.5$.
\end{itemize}

\textbf{Discriminator} $D_t$:
\begin{itemize}
\item It has 2 fully connected layers. The first fully connected layer is followed by a LeakyReLU layer with a negative slope set as 0.2.
\item The input is the concatenation of a target image feature and its multi-label.
\item We use the ADAM optimizer with $\beta=0.5$.
\end{itemize}

\textbf{Classifier $C$}:
\begin{itemize}
\item It consists of one linear softmax layer.
\item We use the ADAM optimizer with $\beta=0.5$.
\end{itemize}

\subsection{On the AwA Dataset}
The parameters of AwA1 and AwA2 are identical.

\textbf{Hyperparameters}:
\begin{itemize}
\item Batch size is 64.
\item Base learning rate is 0.00001.
\item The loss coefficient $\lambda_C$ of pre-trained classifier is 0.01.
\item The loss coefficient $\lambda_t$ of target domain classifier is 1.
\item The loss coefficient $\lambda_a$ of self-supervised learning module is 0.01.
\item Parameters are initialized from zero centered normal distribution with a standard deviation 0.02.
\item The synthesized feature number is 1800.
\item The margin $\mu$ in the triplet mining loss is 0.3.
\item The number of epoch is 300.
\end{itemize}
The architectures of $G, D_s, D_t, C$ are the same as those on the CUB dataset.


\subsection{On the SUN Dataset}
\textbf{Hyperparameters}:
\begin{itemize}
\item Batch size is 64.
\item Base learning rate is 0.0002.
\item The loss coefficient $\lambda_C$ of pre-trained classifier is 0.01.
\item The loss coefficient $\lambda_t$ of target domain classifier is 1.
\item The loss coefficient $\lambda_a$ of self-supervised learning module is 0.01.
\item Parameters are initialized from zero centered normal distribution with a standard deviation 0.02.
\item The synthesized feature number is 400.
\item The margin $\mu$ in the triplet mining loss is 0.3.
\item The number of epoch is 300.
\end{itemize}
The architectures of $G, D_s, D_t, C$ are the same as those on the CUB dataset.


\subsection{On the FLO Dataset}
\textbf{Hyperparameters}:
\begin{itemize}
\item Batch size is 64.
\item Base learning rate is 0.000001.
\item The loss coefficient $\lambda_C$ of pre-trained classifier is 0.01.
\item The loss coefficient $\lambda_t$ of target domain classifier is 1.
\item The loss coefficient $\lambda_a$ of self-supervised learning module is 0.01.
\item Parameters are initialized from zero centered normal distribution with a standard deviation 0.02.
\item The synthesized feature number is 1200.
\item The margin $\mu$ in the triplet mining loss is 0.3.
\item The number of epoch is 300.
\end{itemize}
The architectures of $G, D_s, D_t, C$ are the same as those on the CUB dataset.


\section{More Visualization Results}
Here, we give more t-SNE visualization results on generated features of 
baseline and the proposed SDGN. 
We show 10 groups of comparison results. In each group, the left side is the baseline fake features, while the right side is the SDGN fake features. We randomly choose different source classes and target classes to take a insight into the data distribution of generated features.

\begin{figure}[H]
\centering
\includegraphics[width=1.0\textwidth]{graph/t-sne-d5.pdf}
\label{t-sne2}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=1.0\textwidth]{graph/t-sne-d4.pdf}
\label{t-sne3}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=1.0\textwidth]{graph/t-sne-d3.pdf}
\label{t-sne4}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=1.0\textwidth]{graph/t-sne-d2.pdf}
\label{t-sne4}
\end{figure}



\begin{figure}[H]
\centering
\includegraphics[width=1.0\textwidth]{graph/t-sne-d1.pdf}
\caption{The t-SNE visualization results of generated features of the baseline method (left side) and our proposed SDGN (right side). $\times$ denotes the features of source classes, while $\bullet$ denotes the features of target classes.}
\label{t-sne5}
\end{figure}

As shown in Figure~\ref{t-sne5}, across 10 groups of comparison results 
we can see consistent characteristics of feature distribution of source and target classes. The synthesized features of the baseline method are crowding together, while in our SDGN model, the target features are evidently pushed away
from source features, and features have more segregated clusters
for both source and target domains.

\section{Source Code}
Our source code for testing is shared in folder ``Code-paper5641'',  which includes five files, i.e., `demo.py', `classifier.py', `model.py', `util.py' and `ReadMe.md'.
\begin{enumerate}
\item `demo.py' contains the test code for evaluating the model performance on GZSL and ZSL.
\item `classifier.py' contains the code for constructing the final softmax classifier by use of fake features synthesized by a trained generator. It also provides the specific procedures to calculate the seen accuracy, unseen accuracy and harmonic mean.
\item `model.py' contains the various kinds of model components for the generator and discriminators.
\item `util.py' provides the code for loading data for various datasets.
\item `ReadMe.md' introduces the implementation process in details.
\end{enumerate}

For more details, please refer to the `ReadMe.md'.
%
%{\color{red}
%\cite{junyu2019AAAI_TS-GCN}
%}

{\small
\bibliographystyle{ieee_fullname}
\bibliography{cvpr2020}
}


\end{document}
