kaimingyang/VideoModel_as_RoboPolicy_for_Dream.exe

Dream.exe Wan 2.2 image-to-video LoRA checkpoints

This repository releases the Wan 2.2-LoRA 2K and 7K models described in Dream.exe. They adapt Wan-AI/Wan2.2-I2V-A14B to generate robot-manipulation videos from a first-frame image and a text instruction. These files are adapters, not a standalone or merged Wan model. Download the upstream Wan 2.2 I2V A14B weights separately.

The paper compares Wan 2.2 LoRA variants trained for 2K and 7K optimization steps. This release includes both checkpoint steps and both noise experts; its recorded data and training configuration are specified below.

Repository structure

Wan2.2_I2V_A14B_lora_2k/
├── high_noise_model.safetensors
└── low_noise_model.safetensors
Wan2.2_I2V_A14B_lora_7k/
├── high_noise_model.safetensors
└── low_noise_model.safetensors
training_config.json
inference_config.json
weights_manifest.json

The high- and low-noise adapters at a given step are a pair. Load both with the matching step count. The 2K and 7K pairs are snapshots of the same training lineage, not independent restarts. weights_manifest.json records each file's SHA256 hash, byte size, and base-model revision.

Download one pair with:

hf download kaimingyang/VideoModel_as_RoboPolicy_for_Dream.exe \
  --include 'Wan2.2_I2V_A14B_lora_7k/*' 'inference_config.json' \
  --local-dir ./dream-exe-wan-lora
hf download Wan-AI/Wan2.2-I2V-A14B --local-dir ./Wan2.2-I2V-A14B

Training

Both noise experts were fine-tuned with image, text, and video supervision on all 158 examples in kaimingyang/robovideo_ti2v at dataset revision 7e51aa8eb0c387afbdbdc4daef7afb8f764b490e. The source's train_77.zip and test_81.zip were both used for training. There is no held-out test or validation split for these checkpoints, so the source archive names do not indicate an evaluation protocol here.

At public Dream.exe dataset revision 1211e33baf18ccd1aeff2ad5e1993ad7dbdb2641, 73 of these 158 training case IDs also occur in the 101-case benchmark. Spot checks of first frames confirmed exact or effectively identical images. These released adapters must therefore not be used as held-out models for that benchmark. The paper's disjoint-training statement and reported task-success rates should not be attributed to this particular retraining run.

The LoRA uses rank 32 on q,k,v,o,ffn.0,ffn.2, learning rate 1e-4, 480 × 480 training frames, 49-frame clips, and separate high/low timestep ranges split at 0.358. Both experts reached 7,000 optimizer steps, with 2,000-step snapshots retained. The training implementation is DiffSynth-Studio at commit 7686e54d41d25c0e8ed5f1318acc23b6bb832654; exact recorded settings are in training_config.json. Four GPUs were used, two per expert. The paper specifies the 2K/7K comparison; the remaining recorded hyperparameters follow the pinned DiffSynth Wan 2.2 I2V LoRA recipe and the training setup used here. The pinned loader takes the first 49 frames of each source rollout; it does not resample the complete trajectory into 49 frames. This detail matters when interpreting the generated actions.

Inference

Use a compatible DiffSynth-Studio checkout and the upstream Wan 2.2 I2V A14B model. The following shows the adapter loading and image-conditioned call; inference_config.json records the generation settings used for the released example videos.

from pathlib import Path
import torch
from PIL import Image
from diffsynth.pipelines.wan_video import ModelConfig, WanVideoPipeline
from diffsynth.utils.data import save_video

base = Path("./Wan2.2-I2V-A14B")
adapter = Path("./dream-exe-wan-lora/Wan2.2_I2V_A14B_lora_7k")
pipeline = WanVideoPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(path=[str(p) for p in sorted((base / "high_noise_model").glob("diffusion_pytorch_model-*.safetensors"))]),
        ModelConfig(path=[str(p) for p in sorted((base / "low_noise_model").glob("diffusion_pytorch_model-*.safetensors"))]),
        ModelConfig(path=str(base / "models_t5_umt5-xxl-enc-bf16.pth")),
        ModelConfig(path=str(base / "Wan2.1_VAE.pth")),
    ],
    tokenizer_config=ModelConfig(path=str(base / "google/umt5-xxl")),
)
pipeline.load_lora(pipeline.dit, str(adapter / "high_noise_model.safetensors"), alpha=1)
pipeline.load_lora(pipeline.dit2, str(adapter / "low_noise_model.safetensors"), alpha=1)
with Image.open("first_frame.png") as image, torch.inference_mode():
    frames = pipeline(
        prompt="A robot arm closes the oven door.",
        input_image=image.convert("RGB"),
        height=480, width=480, num_frames=81,
        seed=42, tiled=True, cfg_scale=3.5,
        switch_DiT_boundary=0.9,
        num_inference_steps=40, sigma_shift=5.0,
    )
save_video(frames, "generated.mp4", fps=16, quality=9)

The example generation used 81 frames at 16 FPS for nominal 5-second cases and 129 frames for nominal 8-second cases. It also used the negative prompt and deterministic per-case seeds recorded in inference_config.json. For a closer reproduction, apply those settings to the call above.

Generated examples are published in RoboVideo2Phy gen/ and gen_enhanced/ under wan2.2_lora_2k and wan2.2_lora_7k. The standard and enhanced prompt variants each contain 15 cases per checkpoint.

Limitations

The release is intended for research on video-conditioned robot manipulation. An image-conditioned video is not an executable policy or proof of physical task success. The released example videos were checked for media integrity and first-frame agreement. Consult the Dream.exe dataset and paper for the evaluation protocol and reported results.

The 158 source rollouts have a median duration of 12.17 seconds at 15 FPS, while the 49-frame training prefix covers only 3.2 seconds. Later task-completion states are therefore often absent from this LoRA's training clips. In a diagnostic check, extending one oven-closing generation from 5 to 8 seconds did not close the door. These weights should not be treated as guaranteeing that a requested manipulation finishes within a 5- or 8-second clip.

License

These Dream.exe LoRA adapter files are released under CC BY-NC 4.0, following the project's DVD checkpoint release. The upstream Wan 2.2 base model is not included and is provided by Wan-AI under its own Apache 2.0 terms. Users should review the terms applicable to the upstream model and training data for their intended use.

Citation

If you use these adapters or the associated benchmark, please cite Dream.exe:

@article{zhao2026dreamexe,
  title   = {Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?},
  author  = {Zhao, Rui and Yang, Kaiming and Zhu, Jifeng and Chen, Siyang and Wang, Ziqi and Wu, Weijia and Lin, Kevin Qinghong and Wang, Heng and Shou, Mike Zheng},
  journal = {arXiv preprint arXiv:2606.04811},
  year    = {2026}
}

We thank the Wan and DiffSynth-Studio authors for releasing the base model and training implementation.