# Fast Convergence of DETR with Spatially Modulated Co-Attention

Source: [Fast Convergence of DETR with Spatially Modulated Co-Attention](<https://www.sota2.com/research/paper/68fa691c1e209131cea20501>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:53:51.580Z

Authors: Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, Hongsheng Li

Published: 2021-01-19T03:52:44.000Z

## Abstract

The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However, DETR suffers from its slow convergence. Training DETR \\cite{carion2020end} from scratch needs 500 epochs to achieve a high accuracy. To accelerate its convergence, we propose a simple yet effective scheme for improving the DETR framework, namely Spatially Modulated Co-Attention (SMCA) mechanism. The core idea of SMCA is to conduct regression-aware co-attention in DETR by constraining co-attention responses to be high near initially estimated bounding box locations. Our proposed SMCA increases DETR's convergence speed by replacing the original co-attention mechanism in the decoder while keeping other operations in DETR unchanged. Furthermore, by integrating multi-head and scale-selection attention designs into SMCA, our fully-fledged SMCA can achieve better performance compared to DETR with a dilated convolution-based backbone (45.6 mAP at 108 epochs vs. 43.3 mAP at 500 epochs). We perform extensive ablation studies on COCO dataset to validate the effectiveness of the proposed SMCA.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Object Detection on MS COCO (val)](<https://www.sota2.com/research/sota/object-detection-on-ms-coco-val.md>): Object Detection; MS-COCO (val); mAP: 0.444.

- [Object Detection on COCO 2017 (val)](<https://www.sota2.com/research/sota/object-detection-on-coco-2017-val.md>): Object Detection; COCO 2017 (val); AP: 44.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2101.07448>)

[PDF](<https://arxiv.org/pdf/2101.07448.pdf>)