# SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Source: [SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models](<https://www.sota2.com/research/paper/6a055729dbc836b87d828da5>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-15T05:00:12.430Z

Authors: Vladislav Makarov, Mark Gizetdinov, Dmitry Yudin

Published: 2026-05-13T15:27:41.000Z

## Abstract

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Scene Graph Generation on PSG v1.0 (test)](<https://www.sota2.com/research/sota/scene-graph-generation-on-psg-v1-0-test.md>): Scene Graph Generation; PSG v1.0 (test); Failure Rate: 0.

- [Scene Graph Generation on PVSG IoU threshold = 50 (test)](<https://www.sota2.com/research/sota/scene-graph-generation-on-pvsg-iou-threshold-50-test.md>): Scene Graph Generation; PVSG IoU threshold = 50 (test); Generation Time (s): 1.068.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.13667>)

[PDF](<https://arxiv.org/pdf/2605.13667.pdf>)