## Video4Spatial

### Towards Visuospatial Intelligence with Context-Guided Video Generation

## Overview

TL;DR: We investigate whether video generative models can exhibit visuospatial intelligence, a capability central to human cognition, using only visual data as context.
Your browser does not support the video tag.

## Demo Video

## We sample videos as context and perform instruction-following spatial tasks: video-based object grounding and scene navigation. Our method generates videos that fulfill instructional spatial tasks while remaining geometrically consistent with the provided video context.

## Input Video Context

## Object Grounding

## * Red bbox is generated by the model.

## Instruction: The camera moves smoothly through a bedroom. Finally, it focuses on a guitar in the center of the frame.

## Instruction: The camera moves smoothly through a bedroom. Finally, it focuses on a green plant in the center of the frame.

## Instruction: The camera moves smoothly through a bedroom. Finally, it focuses on a door in the center of the frame (seed 1).

## Instruction: The camera moves smoothly through a bedroom. Finally, it focuses on a door in the center of the frame (seed 2).

## Scene Navigation

## Framework

Our framework is intentionally simple: a standard video diffusion architecture trained only with the diffusion objective. The inputs are a video context (several frames from the same environment) and instructions; the output is a video that completes the instructed spatial task while maintaining scene geometry and temporal coherence.

## Comparisons

## Object Grounding

## Our method faithfully grounds the object present in the context, whereas other methods hallucinate the target object.

## * Wan2.2-5B and Veo3 take 1 frame as context, FramePack uses 105 frames as context, while ours uses 337 frames as context.

## Context

## Wan2.2-5B

## Veo3

## FramePack

## Ours

## Instruction: The camera moves smoothly through a bedroom. Finally, it focuses on a bag on top of the dresser in the center of the frame.

## Context

## Wan2.2-5B

## Veo3

## FramePack

## Ours

## Instruction: The camera moves through a desk area. Finally, it focuses on a monitor in the center of the frame.

## Qualitative comparison of object grounding performance.

## Scene Navigation

## Our method delivers the highest perceptual quality and good camera controllability.

## Qualitative comparison of scene navigation results.

## Ablation Study

## Visualization for ablations on condition frame number, CFG, and auxiliary bbox. The instruction is "The camera moves through an office space. Finally, it focuses on a monitor in the center of the frame."

## Context

## Default

## W/ 45 context frames (infer)

## W/ only the first frame (infer)

## W/o context CFG

## W/o auxiliary bbox

## Out-of-Distribution Generalization

## Our models can generalize to out-of-distribution (OOD) scenarios like outdoor scenes and new categories, performing object grounding and scene navigation. Below are two cases of real scenes scanned by iPone.

## Context

## Instruction (simplified): Tree

## Instruction: Bicycle

## Instruction: Camera Pose

## Results

## Context

## Instruction: Monitor

## Instruction: Window

## Instruction: Camera Pose

## Results

## Failure Cases

## Our method still suffers from artifacts such as temporal discontinuities (row 1) and incorrect grounding (row 2) for some cases. We observe that the model accuracy grows as the repeat times increase.

## Context

## Failure Case

## Success Case

## Instruction: The camera moves smoothly through a kitchen. Finally, it focuses on a sink in the center of the frame.

## Context

## Failure Case

## Success Case

## Instruction: The camera moves from a desk area. Finally, it focuses on a suitcase in the center of the frame.