Situation Reasoning on STBench 200 held-out questions
57.8AccuracyMAGE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MAGEBackbone=Qwen3-8B, Judge=Claude Sonnet 4.6, MAGE Executor Model=Opus2026.05 | 57.8 | 20.8 | |
| MAGEBackbone=Qwen3-8B, Judge=Claude Sonnet 4.6, MAGE Executor Model=Sonnet2026.05 | 54.8 | 20.8 | |
| ReflexionBackbone=Qwen3-8B, Judge=Claude Sonnet 4.6, Prompting Scaffold=Reflexion2026.05 | 37 | — | |
| 0-shot CoTBackbone=Qwen3-8B, Judge=Claude Sonnet 4.6, Prompting Scaffold=0-shot CoT2026.05 | 35.5 | — | |
| ReActBackbone=Qwen3-8B, Judge=Claude Sonnet 4.6, Prompting Scaffold=ReAct2026.05 | 31.8 | — | |
| SC10Backbone=Qwen3-8B, Judge=Claude Sonnet 4.6, Prompting Scaffold=SC10, Samples=102026.05 | 20 | — | |
| 8-shotBackbone=Qwen3-8B, Judge=Claude Sonnet 4.6, Prompting Scaffold=8-shot2026.05 | 12 | — |