# SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis

Source: [SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis](<https://www.sota2.com/research/paper/68fa79591e209131cea7d370>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T10:29:14.605Z

Authors: Ssharvien Kumar Sivakumar, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay

Published: 2025-06-03T17:02:38.000Z

## Abstract

Surgical simulation plays a pivotal role in training novice surgeons, accelerating their learning curve and reducing intra-operative errors. However, conventional simulation tools fall short in providing the necessary photorealism and the variability of human anatomy. In response, current methods are shifting towards generative model-based simulators. Yet, these approaches primarily focus on using increasingly complex conditioning for precise synthesis while neglecting the fine-grained human control aspect. To address this gap, we introduce SG2VID, the first diffusion-based video model that leverages Scene Graphs for both precise video synthesis and fine-grained human control. We demonstrate SG2VID's capabilities across three public datasets featuring cataract and cholecystectomy surgery. While SG2VID outperforms previous methods both qualitatively and quantitatively, it also enables precise synthesis, providing accurate control over tool and anatomy's size and movement, entrance of new tools, as well as the overall scene layout. We qualitatively motivate how SG2VID can be used for generative augmentation and present an experiment demonstrating its ability to improve a downstream phase detection task when the training set is extended with our synthetic videos. Finally, to showcase SG2VID's ability to retain human control, we interact with the Scene Graphs to generate new video samples depicting major yet rare intra-operative irregularities.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Synthesis on Cataract-1k (test)](<https://www.sota2.com/research/sota/video-synthesis-on-cataract-1k-test.md>): Video Synthesis; Cataract-1k (test); FVD: 73.

- [Video Synthesis on Cholec80 (test)](<https://www.sota2.com/research/sota/video-synthesis-on-cholec80-test.md>): Video Synthesis; Cholec80 (test); FVD: 457.3.

- [Scene Graph Alignment on Cataract-1k (test)](<https://www.sota2.com/research/sota/scene-graph-alignment-on-cataract-1k-test.md>): Scene Graph Alignment; Cataract-1k (test); BB IoU: 62.4.

- [Scene Graph Alignment on CATARACTS (test)](<https://www.sota2.com/research/sota/scene-graph-alignment-on-cataracts-test.md>): Scene Graph Alignment; CATARACTS (test); BB IoU: 49.3.

- [Scene Graph Alignment on Cholec80 (test)](<https://www.sota2.com/research/sota/scene-graph-alignment-on-cholec80-test.md>): Scene Graph Alignment; Cholec80 (test); BB IoU: 62.3.

- [Phase Recognition on CATARACTS (train test)](<https://www.sota2.com/research/sota/phase-recognition-on-cataracts-train-test.md>): Phase Recognition; CATARACTS (train test); Accuracy: 80.5.

- [Phase Recognition on Cataract-1k (train test)](<https://www.sota2.com/research/sota/phase-recognition-on-cataract-1k-train-test.md>): Phase Recognition; Cataract-1k (train test); Accuracy: 94.

- [Video Synthesis on CATARACTS (test)](<https://www.sota2.com/research/sota/video-synthesis-on-cataracts-test.md>): Video Synthesis; CATARACTS (test); FVD: 363.8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.03082>)

[PDF](<https://arxiv.org/pdf/2506.03082.pdf>)

[Implementation](<https://www.sota2.com/github.com/MECLabTUDA/SG2VID>)
