Agentic Uncertainty Elicitation on SWE-bench Pro (test) benchmark leaderboard · SOTA2 Research

Agentic Uncertainty Elicitation on SWE-bench Pro (test)

0.68AUROCAggressive (max)

Evaluation Results

Method	Links
Aggressive (max)2026.02		0.68	0.44	0.44	0.41
Aggressive (max)2026.02		0.65	0.46	0.46	0.4
Pre-Execution2026.02		0.64	0.37	0.38	0.34
Adversarial Post-Execution2026.02		0.64	0.2	0.24	0.26
Pre-Execution2026.02		0.62	0.35	0.35	0.33
Average2026.02		0.62	0.37	0.37	0.35
Post-Execution2026.02		0.58	0.39	0.42	0.4
Conservative (min)2026.02		0.57	0.29	0.32	0.32
Adversarial Post-Execution2026.02		0.57	0.4	0.53	0.51
Average2026.02		0.57	0.36	0.36	0.33
Adversarial Post-Execution2026.02		0.55	0.26	0.3	0.31
Post-Execution2026.02		0.55	0.34	0.37	0.36
Conservative (min)2026.02		0.54	0.26	0.31	0.3
Pre-Execution2026.02		0.53	0.77	0.77	0.77
Average2026.02		0.53	0.66	0.66	0.65
Conservative (min)2026.02		0.53	0.54	0.65	0.64
Post-Execution2026.02		0.51	0.55	0.66	0.65
Aggressive (max)2026.02		0.51	0.78	0.78	0.78