# SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning Source: [SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning]() Collection index: [llms.txt]() Data updated: 2025-10-26T05:22:00.744Z Authors: Lin Zhang, Xianfang Zeng, Kangcong Li, Gang Yu, Tao Chen Published: 2025-08-08T08:45:52.000Z ## Abstract We propose SC-Captioner, a reinforcement learning framework that enables the self-correcting capability of image caption models. Our crucial technique lies in the design of the reward function to incentivize accurate caption corrections. Specifically, the predicted and reference captions are decomposed into object, attribute, and relation sets using scene-graph parsing algorithms. We calculate the set difference between sets of initial and self-corrected captions to identify added and removed elements. These elements are matched against the reference sets to calculate correctness bonuses for accurate refinements and mistake punishments for wrong additions and removals, thereby forming the final reward. For image caption quality assessment, we propose a set of metrics refined from CAPTURE that alleviate its incomplete precision evaluation and inefficient relation matching problems. Furthermore, we collect a fine-grained annotated image caption dataset, RefinedCaps, consisting of 6.5K diverse images from COCO dataset. Experiments show that applying SC-Captioner on large visual-language models can generate better image captions across various scenarios, significantly outperforming the direct preference optimization training strategy. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Captioning on DOCCI 500 (test)](): Image Captioning; DOCCI 500 (test); CAPTURE: 57.9. ## Original sources and implementation [Original paper]() [PDF]()