CVPR 2026
We present PhysInOne, the largest dataset addressing the critical scarcity of physically-grounded training data for AI systems.
- 2 million videos generated from 153,810 dynamic 3D scenes
- Covers 71 fundamental physical phenomena in everyday environments, spanning four major domains: Mechanics, Optics, Fluid Dynamics, Magnetism
- Includes 2,231 common objects tailored to daily physical interactions
- Enriched with 623 materials across five categories: plastic, metal, wood, stone, and fabric
- Features 528 diverse 3D backgrounds to ensure realism and environmental variety
- Each scene involves 1–3 physical phenomena, reflecting real-world activities
- Supports complex multi-object interactions, with increasing scene complexity
- Average number of objects per scene: 3.9 (single-physics), 6.3 (double-physics), 7.8 (triple-physics)
- Each scene is captured from 13 viewpoints: 12 static cameras and 1 moving camera
- 3D geometry
- Semantic labels
- Object motion and dynamics
- Physical properties
- Natural-language scene descriptions
- Physics-aware video generation
- Short- and long-term future frame prediction
- Physical property estimation
- Motion transfer
- And more...
| Component | Progress | Status | Notes |
|---|---|---|---|
| Rendered Data - Train | ██████████ 100%(122988/122988) |
Released | Last updated: Aug 21 |
| Rendered Data - Test | ██████████ 100% |
Released | All Leaderboard user inputs released; GT excluded |
| Rendered Data - Val | ░░░░░░░░░░ 1%(103/15411) |
In progress | |
| 3D Assets | ██████████ 100% |
Released | 14,847 Val scenes, 120,660 Train scenes, and Train/Val resource libraries |
| Leaderboard | ██████████ 100% |
Released | Public evaluation inputs for all four tasks; GT excluded |
| PMF | ██████████100% |
Released | |
| Baselines | █████░░░░░50% |
In progress | Last updated: Sept. 24th |
| Data processing | ██████████ 100% |
Released | Camera placement, UE rendering, EXR conversion, and point-cloud generation |
| Resource | Link |
|---|---|
| 📄 Paper | arXiv |
| 🌐 Project Page | vlar-group.github.io/PhysInOne |
| 🤗 Dataset | Hugging Face |
| 🏆 Leaderboard Data | Public evaluation inputs |
| 🧊 3D Assets | PhysicBenchmark project assets |
| 🛠️ Data Processing | Camera, rendering, and post-processing tools |
All user-facing evaluation inputs required by the public Leaderboard have been released for the four benchmark tasks. Ground-truth outputs remain private and are not included in the public download.
| Task | Public package count |
|---|---|
| Video Generation | 75,865 files |
| Future Prediction | 103 scene ZIP archives |
| Physical Properties Estimation | 72 scene ZIP archives and 2 shared support files |
| Motion Transfer | 214 scene ZIP archives |
Clone this repository once; the downloader uses only the Python standard library and reads its maintained manifests directly from the repository:
git clone https://github.com/vLAR-group/PhysInOne.git
cd PhysInOneA byte-identical public mirror of scripts/ is maintained under Utils/scripts/.
During downloads, an interactive terminal displays one live progress bar with file count, transferred and estimated total size, current speed, and ETA. Redirected output automatically switches to periodic plain-text progress updates, while complete per-file records remain available in _download_logs/.
Each row below is a complete command that can be copied directly. The remote Leaderboard folder names remain unchanged, while local task directories are created without spaces.
| Task | Copy command | Local directory |
|---|---|---|
| Video Generation | python scripts/download_data.py --task video-generation --output-dir ./PhysInOne_data |
PhysInOne_data/leaderboard/video_generation/ |
| Future Prediction | python scripts/download_data.py --task future-prediction --output-dir ./PhysInOne_data |
PhysInOne_data/leaderboard/future_prediction/ |
| Physical Properties Estimation | python scripts/download_data.py --task physical-properties-estimation --output-dir ./PhysInOne_data |
PhysInOne_data/leaderboard/physical_properties_estimation/ |
| Motion Transfer | python scripts/download_data.py --task motion-transfer --output-dir ./PhysInOne_data |
PhysInOne_data/leaderboard/motion_transfer/ |
| All four Leaderboard tasks | python scripts/download_data.py --task video-generation future-prediction physical-properties-estimation motion-transfer --output-dir ./PhysInOne_data |
PhysInOne_data/leaderboard/ |
The planned public 3D asset release is now 100% complete. It contains shared Unreal Engine 5.5.4 project files, Scene and Trajectory resources for 14,847 validation scenes and 120,660 training scenes, six validation resource archives, and seven training resource archives. The maintained download contains 4,585 files, including 285 ZIP archives, totaling approximately 139.67 GiB. The 273 validation origins reserved for the Motion Transfer test set are intentionally excluded to prevent evaluation leakage.
The Scene and Trajectory archives are intentionally stored side by side in their respective project folders. They remain separate archives: Scene ZIPs contain Train/<Physics>/<Scene>.umap or Val/<Physics>/<Scene>.umap, while Trajectory ZIPs contain Train/<Physics>/<Scene>_trajectory.uasset or Val/<Physics>/<Scene>_trajectory.uasset. The downloader extracts each ZIP into its current parent, producing Content/PhysInOne/Scenes/{Train,Val}/ and Content/PhysInOne/Trajectories/{Train,Val}/.
The Train Scene and Trajectory series each contain batches 0000–0120; the Val series each contain batches 0000–0014. The final Train batch contains 660 scenes, and the final Val batch contains 847 scenes.
python scripts/download_data.py \
--task 3d_assets \
--output-dir ./PhysInOne_dataThe dependency-free downloader retrieves and assembles the complete currently released project subset. It resumes incomplete files, validates and extracts every released archive in place, writes detailed logs, and creates local directories without spaces. The assembled project is stored at PhysInOne_data/assets/PhysicBenchmark/PhysInOne.uproject.
Common options:
| Option | Purpose |
|---|---|
--task NAME [NAME ...] |
Select one or more public releases. |
--output-dir PATH |
Set a whitespace-free local output root. |
--scene TEXT |
Download matching scene names or six-character IDs. |
--workers N |
Set concurrent downloads. |
--extract / --no-extract |
Control ZIP extraction. |
--delete-zip-after-extract |
Save space after verified extraction. |
--dry-run / --list-only |
Preview counts or exact public URLs. |
See the complete download guide, download script, and task manifests for all parameters and examples. Full rendered-data subsets can be prepared with filter_cases.py and downloaded with download_selected.py.
The public data-processing toolkit is released as ordinary source files under Utils/data_processing, rather than as a ZIP archive. It includes static-camera placement, moving-camera and Level Sequence generation, unattended Unreal Engine rendering, EXR conversion, and multi-view point-cloud generation.
The ten Unreal configuration assets must remain under PhysicBenchmark/Content/PhysInOne/Scripts/, corresponding to /Game/PhysInOne/Scripts/ in Unreal Engine. No Hugging Face account or access token is required.
Due to the large scale of PhysInOne, the rendered data and annotations are split across 16 Hugging Face repositories. Each entry shows the shard size, release status, live all-time downloads, live downloads in the last 30 days, and its repository link.
Combined snapshot (Oct 1, 2026): P01–P16 have 1,233,150 all-time downloads and 408,806 downloads in the last 30 days. Including the main repository, the per-repository sums are 1,280,414 and 442,702.
Download badges query the official Hugging Face API and update automatically. Counts are repository-level download events, not deduplicated users; accessing multiple shards can produce one event in each shard.
The PMF (Physical Motion Fidelity) evaluates video similarity in the frequency domain using 3D FFT-based energy distributions. It is designed for physics-aware video generation, future prediction, and motion transfer tasks in the PhysInOne benchmark.
# Step 1: Install PyTorch first (choose your variant)
# CPU only:
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cpu
# CUDA 12.6:
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126
# Step 2: Install pmf from this repo
pip install git+https://github.com/vLAR-group/PhysInOne.git#subdirectory=pmf#!/usr/bin/env python
"""Test PMF metric with random tensors."""
import torch
from pmf import compute_pmf
def main():
torch.manual_seed(42)
B, T, C, H, W = 1, 81, 3, 128, 128
video_pred = torch.randn(B, T, C, H, W)
video_gt = torch.randn(B, T, C, H, W)
score = compute_pmf(video_pred, video_gt, device='cpu')
# If you want to use gpu, set device='cuda'
# score = compute_pmf(video_pred, video_gt, device='cuda')
if isinstance(score, torch.Tensor):
score = score.item()
print(f"PMF similarity score: {score:.4f}")
if __name__ == "__main__":
main()We provide baseline implementations under the ./baselines directory for your reference. We welcome your feedback, please feel free to contact us if you need anything..
📅 Update Schedule: This section is actively being updated throughout Oct. and Nov.
If you find this work useful, please cite:
@article{zhou2026physinone,
title={PhysInOne: Visual Physics Learning and Reasoning in One Suite},
author={Siyuan Zhou and Hejun Wang and Hu Cheng and Jinxi Li and Dongsheng Wang and Junwei Jiang and Yixiao Jin and Jiayue Huang and Shiwei Mao and Shangjia Liu and Yafei Yang and Hongkang Song and Shenxing Wei and Zihui Zhang and Peng Huang and Shijie Liu and Zhengli Hao and Hao Li and Yitian Li and Wenqi Zhou and Zhihan Zhao and Zongqi He and Hongtao Wen and Shouwang Huang and Peng Yun and Bowen Cheng and Pok Kazaf Fu and Wai Kit Lai and Jiahao Chen and Kaiyuan Wang and Zhixuan Sun and Ziqi Li and Haochen Hu and Di Zhang and Chun Ho Yuen and Bing Wang and Zhihua Wang and Chuhang Zou and Bo Yang},
year={2026},
journal={CVPR}
}This project is licensed under the CC BY-NC-SA 4.0 license.
We would like to express our sincere gratitude to all contributors who participated in human evaluations and data collection efforts.