# Modular Diffusion Models for Structured Visual Recognition

Source: [Modular Diffusion Models for Structured Visual Recognition](<https://www.sota2.com/research/paper/6a3a1363206542251d7c741b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-24T05:00:22.884Z

Authors: Siddhesh Khandelwal, Bj\\"orn Ommer, Leonid Sigal

Published: 2026-06-21T22:38:35.000Z

## Abstract

Traditional supervised methods for structured visual recognition tasks -- such as object detection, segmentation, and scene graph generation -- often produce deterministic, fixed outputs, limiting their ability to capture the inherent uncertainty in complex visual scenes. As a consequence, such point estimates are unable to capture the prediction uncertainty (or multi modality) intrinsic to these problems, often arising from natural ambiguities (e.g., ambiguity in size of partially occluded objects, local ambiguity of exact segmentation boundary, etc.) as well as noise and sparsity of training data. To address this limitation, we present Modular Diffusion Models (MDMs), a simple and novel framework that learns a distribution over structured outputs for a given input image. MDMs decompose the diffusion process into distinct, task-specific modules, each focused on capturing a different aspect of the structured information space, such as object categories, spatial locations, and inter-object relationships. This modular design allows each component to be learned independently, with seamless integration at inference without additional training. Furthermore, the modularity of MDMs enables the diffusion process to easily operate over the heterogeneous output space common in many structured learning tasks (e.g., a continuous bounding boxes and discrete class labels). Experimental results over three distinct structured tasks -- object detection, instance segmentation, and scene graph generation -- highlight the benefits of our proposed framework.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Scene Graph Generation on Visual Genome (test)](<https://www.sota2.com/research/sota/scene-graph-generation-on-visual-genome-test.md>): Scene Graph Generation; Visual Genome (test); R@50: 0.329.

- [Instance Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017-val.md>): Instance Segmentation; COCO 2017 (val); APm: not recorded.

- [Object Detection on MS-COCO 2017 (val)](<https://www.sota2.com/research/sota/object-detection-on-ms-coco-2017-val.md>): Object Detection; MS-COCO 2017 (val); mAP: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.22702>)

[PDF](<https://arxiv.org/pdf/2606.22702.pdf>)
