\section{Proposed Method}

\begin{figure*}[ht!]   % span across both columns   % span across both columns
\centering
\includegraphics[
    width=\textwidth,
    keepaspectratio
]{img/GA7}
\caption{\textbf{Overview of the SPEAR framework.}
Each modality-Sentinel-2, Sentinel-1, PlanetScope, and climate variables is tokenized with spectral, meta, and sensor-aware embeddings (wavelength $\lambda$, bandwidth $\Delta\lambda$, and ground sampling distance). 
Masked tokens are processed by a ViT encoder-decoder under three self-supervised objectives: spectral reconstruction (Spectral-MAE), radar denoising (BYOL-Denoise), and climate variable prediction (Climate-MAE). 
During fusion, pretrained modality-specific CLS tokens are linearly projected and integrated through either concatenation or cross-attention, conditioned on encoded modality-presence indicators. 
Frozen encoders (blue) and trainable fusion heads (red) together produce compact, 32-D pixel-level embeddings applicable to downstream tasks such as land-cover classification, crop-type classification , flood detection, and burn-scar detection.
}
\label{fig:ga-full}
\end{figure*}

Building on the gaps identified in previous studies, we now outline the design and components of the proposed SPEAR framework. The objective is to learn per-pixel embeddings for each sensor modality and fuse them into a unified representation. Let $x$ denote the multispectral reflectance vector, $u$ the climate variables, and $r$ the two-channel SAR backscatter for a given pixel. Three self-supervised encoders are employed: Spectral-MAE for optical data, Climate-MAE for environmental variables, and BYOL-Denoise for radar. Each encoder produces a 32-dimensional embedding ($z_{\text{spec}}$, $z_{\text{clim}}$, $z_{\text{rad}}$), which the fusion head projects into a common latent space and integrates via concatenation or cross-attention.

\subsection{SpectralMAE for multispectral imagery}

Let $x \in \mathbb{R}^C$ denote the per-pixel reflectance vector of $C$ spectral bands
captured by a multispectral sensor (Sentinel-2 or PlanetScope).  
Each band $b$ has a central wavelength $\lambda_b$ and a full width at half maximum (FWHM \cite{ji2025specaware}) $\Delta\lambda_b$.  
Both quantities are normalized to $[0,1]$.

\paragraph{Spectral meta-embedding:}
To encode spectral locality and periodicity, we compute Fourier features
\begin{equation}
\phi_k(\lambda_b)
= [\sin(2\pi k\hat{\lambda}_b),\, \cos(2\pi k\hat{\lambda}_b)], 
\quad k = 1,\dots,K,
\end{equation}
and feed the concatenated vector
$[\hat{\lambda}_b,\,\hat{\Delta\lambda}_b,\,\phi_1(\lambda_b),\dots,\phi_K(\lambda_b)]$
to a lightweight meta-embedding network:
\begin{equation}
m_b = 
\mathrm{MLP}_{\mathrm{meta}}
\!\big([\hat{\lambda}_b, \hat{\Delta\lambda}_b, \phi_1,\dots,\phi_K]\big)
\in \mathbb{R}^D.
\end{equation}
This produces a wavelength-aware embedding $m_b$ that conditions the model on
sensor-specific spectral characteristics.

\paragraph{Tokenization and masking:}
Each scalar reflectance $x_b$ is linearly projected 
$v_b = Wx_b$ and combined with the meta-embedding to form
\begin{equation}
t_b = v_b + m_b.
\end{equation}
The sequence of tokens $T=[t_1,\dots,t_C]\in\mathbb{R}^{C\times D}$ is
prepended with a learnable CLS token and passed to a transformer encoder 
$f_\theta$ with $L$ layers and $H$ heads.  
During pretraining, a random fraction $r$ of the bands is masked:
visible tokens $T_{\mathcal{K}}$ and the CLS token are encoded, 
while a transformer decoder $g_\psi$ reconstructs the masked reflectances:
\begin{equation}
\hat{x}_b = g_\psi(f_\theta([\mathrm{CLS},T_{\mathcal{K}}]), m_b),
\quad b\!\in\!\mathcal{M}.
\end{equation}
% The reconstruction objective is
% \begin{equation}
% \mathcal{L}_{\text{spec}} =
% \frac{1}{|\mathcal{M}|}
% \sum_{b\in\mathcal{M}}
% \ell(\hat{x}_b, x_b),
% \end{equation}
% where $\ell$ is mean absolute or mean squared error.  
The reconstruction objective is defined over the masked spectral bands.
Let $\mathcal{M}$ denote the set of masked band indices. The loss is computed as

\begin{equation}
\mathcal{L}_{\text{spec}} =
\frac{1}{|\mathcal{M}|}
\sum_{b=1}^{C}
\mathbf{1}_{\{b \in \mathcal{M}\}} \,
\ell(\hat{x}_b, x_b),
\end{equation}

where $\ell$ is either the mean absolute error (MAE) or mean squared error (MSE).
The CLS embedding $h_{\mathrm{CLS}}\!\in\!\mathbb{R}^D$ becomes the final spectral descriptor $z_{\mathrm{spec}}$.


\paragraph{Optical Sensors.}
Sentinel-2 provides $C{=}10$ spectral bands and PlanetScope SuperDove $C{=}8$ bands, both trained with a mask ratio of $r_{\text{S2}}{=}0.6$. Band details and metadata are listed in Table~\ref{tab:s2_ps_bands}. The two models share weights in the meta-embedding network for spectral alignment but use separate linear projection layers, $W_{\mathrm{val}}^{(\mathrm{S2})}$ and $W_{\mathrm{val}}^{(\mathrm{P})}$, to account for differing radiometric scales. The resulting CLS embeddings, $z_{\mathrm{S2}}$ and $z_{\mathrm{P}}$, act as sensor-invariant spectral descriptors for downstream fusion.


\subsection{ClimateMAE for environmental variables}
Environmental variables such as land surface temperature (day/night), precipitation, and elevation provide crucial context for interpreting spectral signatures.  Let $u \in \mathbb{R}^{C_c}$ be the vector of climate features.  We augment $u$ with geographic coordinates $(\varphi,\lambda)$ (latitude and longitude in degrees).  To encode geographic periodicity we use sinusoidal embeddings:
\begin{equation}
\begin{aligned}
\mathrm{pos}(\varphi) &= [\sin(2\pi \varphi / 180),\, \cos(2\pi \varphi / 180)],\\
\mathrm{pos}(\lambda) &= [\sin(2\pi \lambda / 180),\, \cos(2\pi \lambda / 180)].
\end{aligned}
\end{equation}
% The four values are projected via a linear layer to a $D$‑dimensional location embedding $e_{\text{loc}}$.  The climate variables are projected to the same dimension using a learnable matrix $W_c$, giving $e_{\text{clim}} = W_c u$.  We concatenate $e_{\text{clim}}$ and $e_{\text{loc}}$ and feed the result to an MLP with dropout to obtain $h_{\text{clim}}$.  As in SpectralMAE, we randomly mask a subset of climate variables and reconstruct them, optimizing
The four values are projected via a linear layer to a $D$‑dimensional location embedding $e_{\text{loc}}$.  The climate variables are projected to the same dimension using a learnable matrix $W_c$, giving $e_{\text{clim}} = W_c u$.  We concatenate $e_{\text{clim}}$ and $e_{\text{loc}}$ and feed the result to an MLP with dropout to obtain $h_{\text{clim}}$.  Masked climate features are zero-filled before encoding, as this empirically stabilized training for low-dimensional inputs, optimizing
\begin{equation}
\mathcal{L}_{\text{climate}} = \frac{1}{|\mathcal{M}_c|}\sum_{j\in \mathcal{M}_c} \ell\bigl(\hat{u}_j, u_j\bigr).
\end{equation}
The embedding $h_{\text{clim}}$ of the dimension $D=32$ is the climate representation $z_{\text{clim}}$.  For consistency, we employ a transformer encoder of depth 6, although with a single token it reduces to a feed‑forward network. The pre ‐ training uses ERA5 reanalysis and remote sensing products matched to each pixel during the training period.  A geospatial split is used to prevent leakage across regions (\(0.9^\circ \times 0.9^\circ\) grid blocks; validation blocks randomly selected 
until \(\approx 10\%\) of samples).

\subsection{BYOL‑Denoise for Sentinel‑1 radar}
Radar backscatter in VV and VH polarizations encodes surface roughness, moisture and structure.  However, SAR images suffer from speckle noise and calibration variability.  We adopt Bootstrap‑Your‑Own‑Latent (BYOL) to learn radar embeddings invariant to such artifacts.  Let $r \in \mathbb{R}^2$ be the log‑scaled VV/VH pair for a pixel.  We generate two augmented views $r_1, r_2$ using: (i) random channel dropout (with probability $p_d$), (ii) multiplicative jitter $r\cdot\exp(\varepsilon)$ with $\varepsilon \sim \mathcal{N}(0,\sigma^2)$, (iii) additive Gaussian noise, and (iv) random scaling.  An online encoder $f$ (a small MLP) maps each view to a latent vector $y_i = f(r_i) \in \mathbb{R}^D$, followed by a projection head $g$.  A target encoder $f'$ with parameters updated by exponential moving average (EMA) produces target representations $y_i'$.  BYOL minimizes the cosine distance between online and target representations:
\begin{equation}
\mathcal{L}_{\text{BYOL}} = 1 - \frac{\langle g(y_1), y_2' \rangle}{\|g(y_1)\|\,\|y_2'\|} + 1 - \frac{\langle g(y_2), y_1' \rangle}{\|g(y_2)\|\,\|y_1'\|}.
\end{equation}
To encourage the encoder to preserve physical information, we add a denoising head $h$ that maps the latent representation back to the clean input: $\hat{r}_i = h(y_i)$.  The denoising loss is an $\ell_2$ penalty $\mathcal{L}_{\text{denoise}} = \|\hat{r}_1 - r\|^2 + \|\hat{r}_2 - r\|^2$.  The total radar loss is
\begin{equation}
\mathcal{L}_{\text{radar}} = \mathcal{L}_{\text{BYOL}} + \alpha \mathcal{L}_{\text{denoise}},
\end{equation}
where $\alpha$ balances invariance and reconstruction.  We schedule the EMA momentum of the target network from 0.985 to 0.996 over training epochs.  The radar encoder outputs a $D=32$ radar embedding $z_{\text{S1}}$.

\subsection{Multi-modal Fusion}
Let $z_{\text{S2}}, z_{\text{P}}, z_{\text{S1}}$, and $z_{\text{Clim}}$ denote the 32-dimensional embeddings from Sentinel-2, PlanetScope, Sentinel-1, and Climate modalities. Each embedding is projected via a modality-specific transformation $P_m: \mathbb{R}^{32} \rightarrow \mathbb{R}^{64}$ using a linear layer with LayerNorm and GELU. Missing modalities are handled through learned \textit{zero-slot tokens} that replace unavailable CLS embeddings, while a binary presence mask informs the transformer to process a fixed set of tokens real or placeholder without retraining or padding. This design ensures stable fusion and consistent outputs under sensor dropout, as validated by the missing-modality results in Table~\ref{tab:fusion_results_combined}. Two fusion strategies are explored: (i) concatenation-based aggregation and (ii) cross-attention-based integration enabling inter-modality interaction.

\subsubsection{Fusion via concatenation}
The simplest fusion concatenates the projected embeddings:
\begin{align}
f = \bigl[
    P_{\text{spec}}(z_{\text{S2}}) \| 
    P_{\text{rad}}(z_{\text{S1}}) \| 
    P_{\text{clim}}(z_{\text{Clim}}) 
    \| \nonumber\\
    P_{\text{planet}}(z_{\text{P}})
\bigr] \in \mathbb{R}^{4d}
\end{align}

Absent modalities contribute zeros.  A shallow classifier (e.g. random forest) is trained on $f$ to predict the downstream labels.  For robustness to missing inputs, a binary presence mask is optionally concatenated to the fused embedding before classification. Each entry in the mask indicates whether the corresponding modality (Sentinel-2, Sentinel-1, PlanetScope, or Climate) is available for that sample.


\subsubsection{Fusion via Cross-Attention}

We adopt a cross-attention fusion module that enables each modality to query
information from all others. Let the projected modality embeddings be
\[
U = [\,u_{\text{S2}},\,u_{\text{S1}},\,u_{\text{Clim}},\,u_{\text{P}}\,]
\in \mathbb{R}^{M \times d},
\]
where missing modalities are represented by zeros.

For each modality $m$, the query is
\begin{equation}
q_m = u_m W_Q, 
\label{eq:query}
\end{equation}
with shared keys/values $K = U W_K,\; V = U W_V$. The cross-attention output is
\begin{equation}
\mathrm{Attn}_m(U)
=
\mathrm{Softmax}\!\left(
\frac{q_m K^\top}{\sqrt{d_k}}
\right)V ,
\label{eq:attn}
\end{equation}
where $d_k = d/H$ and $H$ denotes attention heads. (Multi-head attention follows
the standard formulation.)

Each attended vector is refined via a modality-specific feed-forward network
$\phi_m$ (LayerNorm $\rightarrow$ Linear $\rightarrow$ GELU):
\begin{equation}
\tilde{u}_m 
= 
u_m + \phi_m\!\left(\mathrm{Attn}_m(U)\right).
\label{eq:residual}
\end{equation}

We concatenate the refined outputs
\[
f = 
[\,\tilde{u}_{\text{S2}},\,
  \tilde{u}_{\text{S1}},\,
  \tilde{u}_{\text{Clim}},\,
  \tilde{u}_{\text{P}}\,]
\in \mathbb{R}^{Md},
\]
flattening $f$ to obtain the fused embedding for the classifier. Cross-attention
enables \emph{early fusion}, allowing modalities to reweight each other
(e.g., radar attending to climate in cloudy conditions). A binary modality mask
is appended to $f$ to indicate which modalities are present, ensuring robustness
to missing inputs.



\subsection{Implementation and training details}
Each modality encoder is pre‑trained independently on unlabeled data.  SpectralMAE is trained on millions of Sentinel‑2 and PlanetScope pixels using AdamW with learning rate $1\times 10^{-4}$, weight decay 0.05. Mask ratios of 0.6 are used for Sentinel‑2 and PlanetScope, respectively, and four transformer layers with four attention heads are employed.  ClimateMAE uses a similar optimizer and training schedule; half of the climate variables are masked at random per sample.  BYOL‑Denoise for radar is trained using AdamW with learning rate $3\times 10^{-4}$ for 100 epochs and EMA momentum ramping from 0.985 to 0.996.  After pre‑training, encoders are frozen and only the projection layers and the downstream classifier are trained on labelled data.  In practice we use random forests with 500 trees and balanced class weights for classification tasks, and pixel‑wise classification with connected‑component smoothing for segmentation tasks.
