Multi-turn conversation performance on Code
98.3Avg PerformanceFull
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FullModel=DeepSeek-v3.2-Thinking, Setting=Ideal instructions2026.02 | 98.3 | 95.9 | |
| Experience-Driven MediatorModel=DeepSeek-v3.2-Thinking, Setting=Ambiguous user inputs with Mediator2026.02 | 86.1 | 84.8 | |
| FullModel=GPT-5.2, Setting=Ideal instructions2026.02 | 83.2 | 84 | |
| ShardedModel=DeepSeek-v3.2-Thinking, Setting=Ambiguous user inputs2026.02 | 78.4 | 65.7 | |
| FullModel=GPT-4o-mini, Setting=Ideal instructions2026.02 | 74.2 | 76 | |
| Experience-Driven MediatorModel=GPT-5.2, Setting=Ambiguous user inputs with Mediator2026.02 | 69.1 | 70.1 | |
| Experience-Driven MediatorModel=GPT-4o-mini, Setting=Ambiguous user inputs with Mediator2026.02 | 66.9 | 63.6 | |
| ShardedModel=GPT-4o-mini, Setting=Ambiguous user inputs2026.02 | 51.4 | 57 | |
| ShardedModel=GPT-5.2, Setting=Ambiguous user inputs2026.02 | 39.4 | 44 |