Long-Horizon Evaluation with Simulator Feedback on 10 manifest-defined shared-core domains (de novo)
29Success Rate (1-turn)Sonnet 4.5
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Sonnet 4.5Protocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 29 | 66.5 | 69.5 | 37.5 | 3 | |
| GPT-5.2Protocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 25.6 | 62.5 | 67 | 36.9 | 4.5 | |
| Gemini 3.1 ProProtocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 24 | 57.3 | 60.5 | 33.4 | 3.2 | |
| Opus 4.6Protocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 23.8 | 65.5 | 76 | 41.7 | 10.5 | |
| Sonnet 4.6Protocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 22.9 | 66.5 | 71.5 | 43.6 | 5 | |
| Gemini 2.0 FlashProtocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 19.5 | 43.2 | 60.5 | 23.7 | 17.3 | |
| GPT-4oProtocol=5-turn (3 attempts) / 20-turn (1 attempt)2026.03 | 12.8 | 53 | 55 | 40.2 | 2 |