Fault Attribution on TRAIL SWEBench 31 traces
45.16PrecisionRAFFLES
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| RAFFLESModel=claude-opus-4.6, Context Budget=1M token2026.06 | 45.16 | 25.81 | |
| RAFFLESModel=claude-opus-4.6, Context Budget=200K token2026.06 | 45.16 | 29.03 | |
| RAFFLESModel=claude-opus-4.6, Context Budget=100K token2026.06 | 45.16 | 32.26 | |
| SAFARIModel=claude-opus-4.6, Context Budget=25K token2026.06 | 41.94 | 29.03 | |
| single shotModel=claude-opus-4.6, Context Budget=100K token2026.06 | 35.48 | 25.81 | |
| SAFARIModel=claude-opus-4.6, Context Budget=100K token2026.06 | 35.48 | 19.35 | |
| RAFFLESModel=claude-opus-4.6, Context Budget=25K token2026.06 | 35.48 | 19.35 | |
| single shotModel=claude-opus-4.6, Context Budget=1M token2026.06 | 29.03 | 12.9 | |
| Step-by-stepModel=claude-opus-4.6, Context Budget=1M token2026.06 | 29.03 | 19.35 | |
| single shotModel=claude-opus-4.6, Context Budget=200K token2026.06 | 29.03 | 12.9 | |
| Step-by-stepModel=claude-opus-4.6, Context Budget=200K token2026.06 | 29.03 | 19.35 | |
| Step-by-stepModel=claude-opus-4.6, Context Budget=100K token2026.06 | 29.03 | 19.35 | |
| Step-by-stepModel=claude-opus-4.6, Context Budget=25K token2026.06 | 29.03 | 19.35 | |
| SAFARIModel=claude-opus-4.6, Context Budget=1M token2026.06 | 22.58 | 9.68 | |
| SAFARIModel=claude-opus-4.6, Context Budget=200K token2026.06 | 22.58 | 12.9 | |
| single shotModel=claude-opus-4.6, Context Budget=25K token2026.06 | 9.68 | 0 |