# Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

Source: [Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents](<https://www.sota2.com/research/paper/6a2f871fd33dfb23e783c042>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-16T05:00:05.515Z

Authors: Brendan King, Jeffrey Flanigan

Published: 2026-06-12T00:30:33.000Z

## Abstract

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Agentic Coding on Dialo-SWE-Bench](<https://www.sota2.com/research/sota/agentic-coding-on-dialo-swe-bench.md>): Agentic Coding; Dialo-SWE-Bench; Resolution Rate: 58.8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.13995>)

[PDF](<https://arxiv.org/pdf/2606.13995.pdf>)
