# 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

Source: [4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation](<https://www.sota2.com/research/paper/6948d086423ea42265729f78>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-15T05:00:11.244Z

Authors: Chiao-An Yang, Ryo Hachiuma, Sifei Liu, Subhashree Radhakrishnan, Raymond A. Yeh, Yu-Chiang Frank Wang, Min-Hung Chen

Published: 2025-12-18T19:13:44.000Z

## Abstract

Despite advances in Multimodal LLMs (MLLMs), their ability to reason over 3D structures and temporal dynamics remains limited, constrained by weak 4D perception and temporal understanding. Existing 3D and 4D Video Question Answering (VQA) benchmarks also emphasize static scenes and lack region-level prompting. We tackle these issues by introducing: (a) 4D-RGPT, a specialized MLLM designed to capture 4D representations from video inputs with enhanced temporal perception; (b) Perceptual 4D Distillation (P4D), a training framework that transfers 4D representations from a frozen expert model into 4D-RGPT for comprehensive 4D perception; and (c) R4D-Bench, a benchmark for depth-aware dynamic scenes with region-level prompting, built via a hybrid automated and human-verified pipeline. Our 4D-RGPT achieves notable improvements on both existing 4D VQA benchmarks and the proposed R4D-Bench benchmark.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [3D/4D Visual Question Answering on STI Bench 1.0 (test)](<https://www.sota2.com/research/sota/3d-4d-visual-question-answering-on-sti-bench-1-0-test.md>): 3D/4D Visual Question Answering; STI Bench 1.0 (test); Accuracy: 59.1.

- [3D/4D Video Question Answering on VSTI-Bench](<https://www.sota2.com/research/sota/3d-4d-video-question-answering-on-vsti-bench.md>): 3D/4D Video Question Answering; VSTI-Bench; Accuracy: 59.1.

- [3D/4D Visual Question Answering on VLM4D-real 1.0 (test)](<https://www.sota2.com/research/sota/3d-4d-visual-question-answering-on-vlm4d-real-1-0-test.md>): 3D/4D Visual Question Answering; VLM4D-real 1.0 (test); Accuracy (MCQ): 53.7.

- [3D/4D Visual Question Answering on MMSI Bench 1.0 (test)](<https://www.sota2.com/research/sota/3d-4d-visual-question-answering-on-mmsi-bench-1-0-test.md>): 3D/4D Visual Question Answering; MMSI Bench 1.0 (test); Avg Multiple Choice Accuracy: 33.3.

- [3D/4D Visual Question Answering on OmniSpatial 1.0 (test)](<https://www.sota2.com/research/sota/3d-4d-visual-question-answering-on-omnispatial-1-0-test.md>): 3D/4D Visual Question Answering; OmniSpatial 1.0 (test); Accuracy: 42.7.

- [Visual Question Answering on SAT 1.0 (test)](<https://www.sota2.com/research/sota/visual-question-answering-on-sat-1-0-test.md>): Visual Question Answering; SAT 1.0 (test); Accuracy (MCQ): 65.3.

- [4D Understanding on R4D-Bench (All)](<https://www.sota2.com/research/sota/4d-understanding-on-r4d-bench-all.md>): 4D Understanding; R4D-Bench (All); Average Score: 42.2.

- [3D/4D Video Question Answering on SAT](<https://www.sota2.com/research/sota/3d-4d-video-question-answering-on-sat.md>): 3D/4D Video Question Answering; SAT; Accuracy: 64.7.

- [3D/4D Video Question Answering on STI-Bench](<https://www.sota2.com/research/sota/3d-4d-video-question-answering-on-sti-bench.md>): 3D/4D Video Question Answering; STI-Bench; Accuracy: 37.6.

- [3D/4D Video Question Answering on MMSI-Bench](<https://www.sota2.com/research/sota/3d-4d-video-question-answering-on-mmsi-bench.md>): 3D/4D Video Question Answering; MMSI-Bench; Accuracy: 33.3.

- [3D/4D Video Question Answering on VLM4D real](<https://www.sota2.com/research/sota/3d-4d-video-question-answering-on-vlm4d-real.md>): 3D/4D Video Question Answering; VLM4D real; Accuracy: 52.7.

- [3D/4D Video Question Answering on OmniSpatial](<https://www.sota2.com/research/sota/3d-4d-video-question-answering-on-omnispatial.md>): 3D/4D Video Question Answering; OmniSpatial; Accuracy: 40.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2512.17012>)

[PDF](<https://arxiv.org/pdf/2512.17012.pdf>)