As illustrated in Fig. 2, SpatialVAM employs a unified framework for joint spatio-temporal modeling and action prediction. The overall framework can be decomposed into three main components: multi-view representation and projection, joint video and heatmap diffusion, and hierarchical action decoding.

Multi-View Representation and Projection

SpatialVAM transforms input colored point clouds and robot end-effector poses into a format compatible with video foundation models. Specifically, it uses orthographic projection to generate multi-view RGB images from the point cloud and represents robot states as multi-view Gaussian heatmaps. This design implicitly encodes 3D spatial structure while aligning with the representation used in video pre-training.

Joint Video and Heatmap Diffusion

The model leverages a 5B-parameter video foundation model (Wan2.2) augmented with view-attention modules to ensure multi-view consistency. It is trained to jointly predict future multi-view RGB video sequences and corresponding heatmap sequences. By modeling how the environment evolves as a consequence of actions, the model captures environmental dynamics and future end-effector trajectories within a shared latent space.

Hierarchical Action Decoding

To recover executable actions, SpatialVAM employs a two-stream decoding process. The predicted heatmap peaks are back-projected into 3D space using camera parameters to recover a continuous 3D end-effector trajectory. Simultaneously, a lightweight rotation and gripper predictor takes the denoised video latents and localized heatmap features as input to estimate the end-effector's rotation (Euler angles) and gripper states.

Figure 2: Network architecture of SpatialVAM

For the inference denoising steps N, we vary it from 1 to 50. The results are presented in Fig. 3. Surprisingly, in contrast to typical video diffusion processes, which require around 50 denoising steps, SpatialVAM achieves a comparable success rate with just a single denoising step. We believe the main reason for this is that heatmaps have relatively simple distribution modes and lack the high-frequency details present in regular images. Furthermore, our action prediction relies solely on the peak locations of the heatmaps, meaning that the overall quality of the heatmaps is less critical, and thus fewer denoising steps are required.

Figure 3: Average success rates for different inference denoising steps. SpatialVAM demonstrates high robustness to varying diffusion steps, achieving strong performance even when the denoising step is set to 1.

For better visualization, we present example prediction results with different inference denoising steps in Fig. 4. As shown, when the denoising step is set to 1, the visual quality of the RGB prediction is somewhat lower, but the predicted heatmaps still provide meaningful information.

❮ ❯

Figure 4: Visualization of video predictions under different denoising steps. Predicted RGB videos and heatmap videos under different denoising step settings. Lower denoising steps lead to visibly lower RGB video quality, while the predicted heatmaps remain relatively stable.

We conduct extensive ablation studies on the Meta-World benchmark to validate the key design choices in SpatialVAM, as summarized in Tab. 4 (rows ordered by success rate). Model #2 applies full fine-tuning instead of LoRA and reaches 87.4% — comparable to LoRA but with substantially higher computational and memory costs — so we adopt LoRA throughout.

Model #3 concatenates heatmaps and RGB frames along the channel dimension (with an additional convolutional adapter to match the DiT's input channels) rather than the view dimension. The resulting information bottleneck drops performance to 81.1%, confirming that view-dimension concatenation better preserves multi-view information.

Model #4 bypasses heatmap back-projection and regresses the 3D translation directly from the denoised latents with a transformer; the success rate falls markedly to 76.6%, validating that heatmap-peak back-projection is substantially more effective than latent-space regression for translation decoding. Model #5 removes the pretrained video-foundation-model weights and almost completely fails (4.6%), as the model cannot even fit the training set, underscoring the essential role of large-scale video pretraining for data-efficient manipulation.

Finally, we vary the number of additionally predicted RGB views from 0 to 3, as shown in Fig. 4. The success rate grows monotonically from 61.1% to 89.1%, showing that (i) jointly predicting RGB and heatmap videos is markedly better than predicting heatmaps alone, and (ii) predicting more views yields a more complete spatio-temporal understanding and better downstream control.

Figure 4: Average success rate vs. the number of additionally predicted RGB views. Predicting more views yields monotonic improvements, from 61.1% with heatmap-only prediction to 89.1% with three additional RGB views.

Deploying manipulation policies in the real world poses safety challenges: it's difficult to assess whether a predicted action sequence is reasonable or safe from its raw numerical representation. In practice, reliable verification often requires actual action execution, which can be unsafe and may damage the robot or surroundings.

In contrast, SpatialVAM generates realistic, temporally consistent multi-view video and heatmap sequences, allowing users to visually inspect predicted rollouts before execution. This provides a safer, more interpretable action validation mechanism.

To quantify this benefit, we conducted a study with four evaluators, each performing 35 rollouts (140 in total). The evaluators reviewed the generated videos before execution and reran any rollout that appeared unsafe (e.g., potential collisions). Tab. 4 shows that video-based inspection significantly reduces collision events, demonstrating that SpatialVAM's predicted RGB videos enhance the interpretability of action outputs and provide a practical tool for safer deployment.