
%
\section{Proposed Methodology}
\label{sec:formatting}
\begin{figure*}[h]
	\centering
	\includegraphics[width=1.0\textwidth]{Multimodal Dual-domain learning for image fusion/figure/overview1.pdf}
	\caption{Overview of the $i$-th recurrent unit in the proposed DDLN framework.}
	\label{overview}
\end{figure*}

\subsection{Overall Framework}
Given a LR hyperspectral image $\bm{\mathcal{X}} \in \mathbb{R}^{B \times w \times h}$ and a corresponding HR multispectral image $\bm{\mathcal{Y}} \in \mathbb{R}^{b \times W \times H}$, the objective of this work is to reconstruct a high-fidelity super-resolved hyperspectral image $\bm{\mathcal{Z}} \in \mathbb{R}^{B \times W \times H}$ using the proposed Dual-Domain Learning Network (DDLN), as illustrated in Fig. \ref{overview}. Here, $(w, h)$ and $(W, H)$ denote the spatial dimensions, while $B$ and $b$ represent the number of spectral bands in the hyperspectral and multispectral images, respectively, with $w \ll W$, $h \ll H$, and $b \ll B$.

As a recursive architecture, DDLN performs progressive band-wise reconstruction by leveraging both spatial and spectral correlations. To exploit the spectral redundancy across adjacent bands, the LR hyperspectral cube $\bm{\mathcal{X}}$ is divided into $B$ spectral groups, denoted by ${ \bm{G}_1, \dots, \bm{G}_i, \dots, \bm{G}_B }$, where the $i$-th group is defined as:
\begin{equation}
	\bm{G}_{i} = [ {\bm{I}_{i-2}^{lr}, \bm{I}_{i-1}^{lr}, \bm{I}_i^{lr}, \bm{I}_{i+1}^{lr}, \bm{I}_{i+2}^{lr}}] ,  3\le i \le B-2 ,
\end{equation}
where $\bm{I}^{lr}_i$ is the $i$-th spectral band of $\bm{\mathcal{X}}$. For edge bands (i.e., $i \in {1, 2, B-1, B}$), circular padding is employed to maintain group consistency.

To reconstruct a target band $\bm{I}^{lr}_i$, the model takes as input both its corresponding spectral group $\bm{G}_i$ and the HR multispectral image $\bm{\mathcal{Y}}$. As shown in Fig. \ref{overview}, these two modalities are processed in parallel branches to exploit their complementary information. To align spatial dimensions with the HR guidance, the hyperspectral group $\bm{G}_i$ is first upsampled using bicubic interpolation, denoted as $\bm{G}_i \uparrow$. Furthermore, the super-resolved output from the previous band $\bm{I}^{sr}{i-1}$ is concatenated with $\bm{G}_i \uparrow$ to enhance spatial and spectral coherence. The shallow feature extraction for the two inputs is defined as:
\begin{equation}
	\bm H_0 = Conv(Cat(\bm{G}_{i} \uparrow, \bm{I}_{i-1}^{sr})) ,
\end{equation}
\begin{equation}
	\bm M_0 = Conv(\bm{\mathcal{Y}}) ,
\end{equation}
where $Conv(\cdot)$ denotes the convolution operation, $Cat(\cdot)$ is the concatenation operation, and $\bm{H}_0$ and $\bm{M}_0$ represent the extracted low-level features for the hyperspectral and multispectral branches, respectively.

To facilitate effective cross-modal feature fusion and enhance structural detail recovery, a sequence of $K$ dual-domain fusion (DDF) modules is employed. These modules iteratively refine both feature streams, focusing on capturing complementary information in the frequency and spatial domains:
\begin{equation}
	\bm H_K, \bm M_K = \mathcal{M}_{K}(...\mathcal{M}_{k}(...\mathcal{M}_{1}(\bm H_0, \bm M_0)...)...) ,
\end{equation}
where $\mathcal{M}_{k}$ denotes the $k$-th DDF module, and $(\bm{H}_K, \bm{M}_K)$ are the final enhanced features passed to the subsequent reconstruction stage. 

Subsequently, the reconstruction feature for the $i$-th spectral band, denoted as $\bm{F}_{RF}^{i}$, is derived by adaptively aggregating multi-level features ($\bm{H}_1$, $\bm{H}_2$, ..., $\bm{H}_K$) together with the spectral context from the previous band $\bm{F}_{RF}^{i-1}$. This process is formulated as:
\begin{equation}
	\scalebox{0.95}{$\bm{F}_{RF}^{i} = Conv(Cat(w_1\bm H_1,...,w_K\bm H_K, w_{K+1} \bm{F}_{RF}^{i-1}))$} ,
\end{equation}
where $w_1$, $w_2$,..., $w_{K+1}$ are learnable weights that are adaptively tuned during training to balance the contributions from different sources.

To produce the super-resolved output $\bm{I}_i^{sr}$, global residual learning is employed by combining the aggregated feature $\bm{F}_{RF}^{i-1}$ with the upsampled LR input band $\bm{I}_i^{lr}\uparrow$, as follows:
\begin{equation}
	\bm{I}_{i}^{sr} = Conv(\bm{F}_{RF}^{i-1}+Conv(\bm{I}_i^{lr}\uparrow)) ,
\end{equation}
where $\bm{I}_i^{lr}\uparrow$ denotes the bicubic-interpolated version of the $i$-th LR spectral band.

After iteratively processing all spectral bands, the final super-resolved hyperspectral cube $\bm{\mathcal{Z}}$ is obtained by concatenating the outputs of all bands as follows:
\begin{equation}
	\bm{\mathcal{Z}} = [\bm{I}_{1}^{sr},...,\bm{I}_{i}^{sr},...,\bm{I}_{B}^{sr}] .
\end{equation}

\subsection{Dual-Domain Fusion Module}

Fig. \ref{overview} illustrates the architecture of the proposed DDF module. By taking the first DDF module as an example, the frequency-domain branch employs the discrete fourier transform (DFT) to effectively extract and integrate frequency features from the input hyperspectral and multispectral modalities ($\bm{H}_0$ and $\bm{M}_0$), thus resulting in a unified frequency representation $\bm{F}_1^{Fre}$:
\begin{equation}
	\bm{F}_1^{Fre} = \mathcal B_1^{Fre}(\bm{H}_0, \bm{M}_0),
\end{equation}
where $\mathcal B_1^{Fre}$ denotes the frequency-domain branch of the $1$-th DDF module. Simultaneously, the spatial-domain branch explores spatial features to generate complementary multimodal spatial information $\bm{F}_1^{Spa}$: 
\begin{equation}
	\bm{F}_1^{Spa} = \mathcal B_1^{Spa}(\bm{H}_0, \bm{M}_0),
\end{equation}
where $\mathcal B_1^{Spa}$ represents the spatial-domain branch of the $1$-th DDF module. The frequency and spatial features are then fused to produce dual-domain joint representation $\bm F_1^{Dual}$:
\begin{equation}
	\bm F_1^{Dual} = Conv(Cat(\bm{F}_1^{Fre}, \bm{F}_1^{Spa})),
\end{equation}
where $Cat(\cdot)$ and $Conv(\cdot)$ are concatenation and convolution operations. Finally, this joint representation is combined with the original inputs to produce enhanced modality-specific features:
\begin{equation}
	\bm H_1 = \bm H_1+\bm F_1^{Dual}, \bm M_1 = \bm M_1+\bm F_1^{Dual}.
\end{equation}

By integrating spatial and frequency cues, the DDF module enhances the representational power of each modality, supporting more accurate information reconstruction. In the following sections, we provide a detailed description of the frequency domain and spatial domain branches.

\subsubsection{Frequency Domain Branch}
To explore both modality-specific and complementary frequency information, the frequency domain branch operates in the Fourier domain, where the hyperspectral and multispectral features are decomposed into their corresponding amplitude and phase components. Given the input modality-specific features $\bm{H}_0$ and $\bm{M}_0$ from the hyperspectral and multispectral modalities, respectively,  we perform the DFT to extract their frequency-domain representations:
\begin{equation}
\bm{A}_1^H, \bm{P}_1^H = DFT(\bm{H}_0),
\end{equation}
\begin{equation}
\bm{A}_1^M, \bm{P}_1^M = DFT(\bm{M}_0),
\end{equation}
where the $DFT(\cdot)$ denotes the discrete fourier transform, $\bm{A}_1^H$ and $\bm{A}_1^M$ denote the amplitude components, while $\bm{P}_1^H$ and $\bm{P}_1^M$ denote the phase components of the hyperspectral and multispectral features, respectively. Then, to effectively fuse the frequency information from different modalities, we adopt differentiated fusion strategies for the amplitude and phase components, based on their distinct representational roles in the Fourier domain. Specifically, the amplitude components encode global structural and semantic information, exhibiting significant inter-channel variation. Therefore, after concatenating the amplitude features from both modalities, we introduce a channel attention mechanism to perform adaptive channel-wise reweighting. This process can be formulated as:
\begin{equation}
      \hat{\bm A}_1 = Conv(Cat(Res(\bm{A}_1^H),Res(\bm{A}_1^M))),
\end{equation}
\begin{equation}
      \bm F_1^{Amp} = \hat{\bm A}_1 \cdot \sigma(Conv_{1\times1}(GAP(\hat{\bm A}_1))),
\end{equation}
where $Res(\cdot)$ is the residual module, $GAP(\cdot)$ denotes global average pooling, $\text{Conv}_{1\times1}(\cdot)$ represents a $1 \times 1$ convolutional layer for channel transformation, and $\sigma(\cdot)$ represents the sigmoid activation. $\bm F_1^{Amp}$ denotes the fused amplitude information. This design allows the network to emphasize informative frequency responses while suppressing redundant or modality-specific noise, thereby enhancing the discriminative capacity of the fused amplitude representation. 

In contrast, the phase components primarily capture fine-grained spatial alignment and texture continuity. To preserve the integrity of the delicate spatial information, we adopt a lightweight fusion strategy. Specifically, the multimodal phase features, $\bm{P}_1^H$ and $\bm{P}_1^M$, are concatenated and subsequently passed through several cascaded convolutional layers:
\begin{equation}
      \bm F_1^{Pha} = Conv^{\times4}(Cat(\bm{P}_1^H, \bm{P}_1^M)),
\end{equation}
where $\bm F_1^{Pha}$ represents the fused phase information and $Conv^{\times4}(\cdot)$ is a stack of four convolutional layers. Subsequently, an inverse Discrete Fourier Transform (iDFT) is applied to the fused amplitude $\bm F_1^{Amp}$ and phase $\bm F_1^{Pha}$ components to project them back to the spatial domain, resulting in the multimodal frequency-domain feature $\bm{F}_1^{Fre}$: 
\begin{equation}
      \bm{F}_1^{Fre} = iDFT(\bm F_1^{Amp},\bm F_1^{Pha}).
\end{equation}

\subsubsection{Spatial Domain Branch}
To enable effective structural collaboration between the hyperspectral and multispectral modalities, the spatial domain branch is designed to simultaneously recalibrate modality-specific features and facilitate cross-modality information enhancement. Given the distinct spatial characteristics of different modalities, including disparities in edge structures and texture patterns, naively combining them may result in the propagation of noise or the accumulation of redundant information. To mitigate this, we first perform independent channel-wise recalibration for each modality to emphasize its most informative features. This is followed by cross-modality enhancement, which promotes accurate spatial alignment and structural consistency between the two modalities.

Specifically, the input multimodal features $\bm{H}_0$ and $\bm{M}_0$ are first passed through independent channel attention mechanisms, aiming to capture global spatial context and generate channel-wise importance weights ($\bm{a}_1^H$ and $\bm{a}_1^M$): 
%each consisting of a global average pooling, a convolution, and a multilayer perceptron (MLP). This operation captures global spatial context and generates channel-wise importance weights:
\begin{equation}
\bm{a}_1^H = \sigma({MLP}({Conv}_{1\times1}({GAP}(\bm{H}_0)))),
\end{equation}
\begin{equation}
  \quad \quad \bm{a}_1^M = \sigma({MLP}({Conv}_{1\times1}({GAP}(\bm{M}_0)))),
\end{equation}
where $MLP(\cdot)$ refers to a multilayer perceptron network. Then, the resulting attention vectors are used to recalibrate each modality via channel-wise multiplication:
\begin{equation}
\bm{H}_1' = \bm{H}_0 \cdot \bm{a}_1^H, \quad
\bm{M}_1' = \bm{M}_0 \cdot \bm{a}_1^M.
\end{equation}
To capture complementary spatial cues, we employ a cross-modality enhancement strategy, where the recalibrated features from one modality are added to those of the other, generating modality-specific representations that incorporate complementary information:
\begin{equation}
\tilde{\bm{H}} = \bm{H}_0 + \bm{M}_1', \quad
\tilde{\bm{M}} = \bm{M}_0 + \bm{H}_1'.
\end{equation}
These enhanced features are then concatenated along the channel dimension and integrated via a convolution operation, producing the complementary multimodal information in the spatial domain:
\begin{equation}
\bm{F}_1^{Spa} = {Conv}_{1\times1}(\text{Cat}(\tilde{\bm{H}}, \tilde{\bm{M}})).
\end{equation}
This design not only improves the structural expressiveness within each modality but also ensures complementary information exchange across modalities. The spatial domain representation $\bm{F}_1^{Spa}$ thus serves as a robust structural foundation for subsequent dual-domain fusion.



% All text must be in a two-column format.
% The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
% Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
% The main title (on the first page) should begin 1 inch (2.54 cm) from the top edge of the page.
% The second and following pages should begin 1 inch (2.54 cm) from the top edge.
% On all pages, the bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
% for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the
% page.

% %-------------------------------------------------------------------------
% \subsection{Margins and page numbering}

% All printed material, including text, illustrations, and charts, must be kept
% within a print area $6\frac{7}{8}$ inches (17.46 cm) wide by $8\frac{7}{8}$ inches (22.54 cm)
% high.
% %
% Page numbers should be in the footer, centered and $\frac{3}{4}$ inches from the bottom of the page.
% The review version should have page numbers, yet the final version submitted as camera ready should not show any page numbers.
% The \LaTeX\ template takes care of this when used properly.



% %-------------------------------------------------------------------------
% \subsection{Type style and fonts}

% Wherever Times is specified, Times Roman may also be used.
% If neither is available on your word processor, please use the font closest in
% appearance to Times to which you have access.

% MAIN TITLE.
% Center the title $1\frac{3}{8}$ inches (3.49 cm) from the top edge of the first page.
% The title should be in Times 14-point, boldface type.
% Capitalize the first letter of nouns, pronouns, verbs, adjectives, and adverbs;
% do not capitalize articles, coordinate conjunctions, or prepositions (unless the title begins with such a word).
% Leave two blank lines after the title.

% AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
% and printed in Times 12-point, non-boldface type.
% This information is to be followed by two blank lines.

% The ABSTRACT and MAIN TEXT are to be in a two-column format.

% MAIN TEXT.
% Type main text in 10-point Times, single-spaced.
% Do NOT use double-spacing.
% All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
% Make sure your text is fully justified---that is, flush left and flush right.
% Please do not place any additional blank lines between paragraphs.

% Figure and table captions should be 9-point Roman type as in \cref{fig:onecol,fig:short}.
% Short captions should be centred.

% \noindent Callouts should be 9-point Helvetica, non-boldface type.
% Initially capitalize only the first word of section titles and first-, second-, and third-order headings.

% FIRST-ORDER HEADINGS.
% (For example, {\large \bf 1. Introduction}) should be Times 12-point boldface, initially capitalized, flush left, with one blank line before, and one blank line after.

% SECOND-ORDER HEADINGS.
% (For example, { \bf 1.1. Database elements}) should be Times 11-point boldface, initially capitalized, flush left, with one blank line before, and one after.
% If you require a third-order heading (we discourage it), use 10-point Times, boldface, initially capitalized, flush left, preceded by one blank line, followed by a period and your text on the same line.

% %-------------------------------------------------------------------------
% \subsection{Footnotes}

% Please use footnotes\footnote{This is what a footnote looks like.
% It often distracts the reader from the main flow of the argument.} sparingly.
% Indeed, try to avoid footnotes altogether and include necessary peripheral observations in the text (within parentheses, if you prefer, as in this sentence).
% If you wish to use a footnote, place it at the bottom of the column on the page on which it is referenced.
% Use Times 8-point type, single-spaced.


% %-------------------------------------------------------------------------
% \subsection{Cross-references}

% For the benefit of author(s) and readers, please use the
% {\small\begin{verbatim}
%   \cref{...}
% \end{verbatim}}  command for cross-referencing to figures, tables, equations, or sections.
% This will automatically insert the appropriate label alongside the cross-reference as in this example:
% \begin{quotation}
%   To see how our method outperforms previous work, please see \cref{fig:onecol} and \cref{tab:example}.
%   It is also possible to refer to multiple targets as once, \eg~to \cref{fig:onecol,fig:short-a}.
%   You may also return to \cref{sec:formatting} or look at \cref{eq:also-important}.
% \end{quotation}
% If you do not wish to abbreviate the label, for example at the beginning of the sentence, you can use the
% {\small\begin{verbatim}
%   \Cref{...}
% \end{verbatim}}
% command. Here is an example:
% \begin{quotation}
%   \Cref{fig:onecol} is also quite important.
% \end{quotation}

% %-------------------------------------------------------------------------
% \subsection{References}

% List and number all bibliographical references in 9-point Times, single-spaced, at the end of your paper.
% When referenced in the text, enclose the citation number in square brackets, for
% example~\cite{Authors14}.
% Where appropriate, include page numbers and the name(s) of editors of referenced books.
% When you cite multiple papers at once, please make sure that you cite them in numerical order like this \cite{Alpher02,Alpher03,Alpher05,Authors14b,Authors14}.
% If you use the template as advised, this will be taken care of automatically.

% \begin{table}
%   \centering
%   \begin{tabular}{@{}lc@{}}
%     \toprule
%     Method & Frobnability \\
%     \midrule
%     Theirs & Frumpy \\
%     Yours & Frobbly \\
%     Ours & Makes one's heart Frob\\
%     \bottomrule
%   \end{tabular}
%   \caption{Results.   Ours is better.}
%   \label{tab:example}
% \end{table}

% %-------------------------------------------------------------------------
% \subsection{Illustrations, graphs, and photographs}

% All graphics should be centered.
% In \LaTeX, avoid using the \texttt{center} environment for this purpose, as this adds potentially unwanted whitespace.
% Instead use
% {\small\begin{verbatim}
%   \centering
% \end{verbatim}}
% at the beginning of your figure.
% Please ensure that any point you wish to make is resolvable in a printed copy of the paper.
% Resize fonts in figures to match the font in the body text, and choose line widths that render effectively in print.
% Readers (and reviewers), even of an electronic copy, may choose to print your paper in order to read it.
% You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

% When placing figures in \LaTeX, it's almost always best to use \verb+\includegraphics+, and to specify the figure width as a multiple of the line width as in the example below
% {\small\begin{verbatim}
%    \usepackage{graphicx} ...
%    \includegraphics[width=0.8\linewidth]
%                    {myfile.pdf}
% \end{verbatim}
% }


% %-------------------------------------------------------------------------
% \subsection{Color}

% Please refer to the author guidelines on the \confName\ \confYear\ web page for a discussion of the use of color in your document.

% If you use color in your plots, please keep in mind that a significant subset of reviewers and readers may have a color vision deficiency; red-green blindness is the most frequent kind.
% Hence avoid relying only on color as the discriminative feature in plots (such as red \vs green lines), but add a second discriminative feature to ease disambiguation.