Theory of Mind on BigToM
99.5AccuracyRECTOM
Evaluation Results
| Method | Links | |
|---|---|---|
| RECTOMModel=GPT-5.42026.06 | 99.5 | |
| CoTModel=GPT-5.42026.06 | 99 | |
| SIMTOMModel=GPT-5.42026.06 | 99 | |
| RECTOMModel=Gemini-32026.06 | 99 | |
| RECTOMModel=Gemma-42026.06 | 99 | |
| CoTModel=Gemini-32026.06 | 98.75 | |
| UserHarnessBackbone=Claude-Opus-4.72026.05 | 98.67 | |
| RECTOMModel=Qwen3.52026.06 | 98.5 | |
| Ledger promptingBackbone=GPT-5.42026.05 | 98.25 | |
| UserHarnessBackbone=Gemini-3.1-Pro2026.05 | 98.25 | |
| UserHarnessBackbone=Claude-Sonnet-4.62026.05 | 98.17 | |
| SIMTOMModel=Gemini-32026.06 | 98 | |
| TIMETOMModel=GPT-5.42026.06 | 97.75 | |
| UserHarnessBackbone=GPT-5.42026.05 | 97.67 | |
| Ledger promptingBackbone=Qwen3-14B2026.05 | 97.25 | |
| Ledger promptingBackbone=GPT-OSS-120B2026.05 | 97.08 | |
| Ledger promptingBackbone=GPT-5.4-mini2026.05 | 96.67 | |
| Ledger promptingBackbone=Qwen3-32B2026.05 | 96.5 | |
| CoTModel=Qwen3.52026.06 | 96.25 | |
| Ledger promptingBackbone=GPT-OSS-20B2026.05 | 96.17 | |
| UserHarnessBackbone=Qwen3-32B2026.05 | 95.42 | |
| UserHarnessBackbone=GPT-5.4-mini2026.05 | 95.25 | |
| SIMTOMModel=Qwen3.52026.06 | 94.75 | |
| UserHarnessBackbone=Qwen3-14B2026.05 | 94.17 | |
| Paradox promptingBackbone=GPT-5.42026.05 | 93.83 | |
| UserHarnessBackbone=GPT-OSS-120B2026.05 | 93.83 | |
| Ledger promptingBackbone=Qwen3-8B2026.05 | 93.5 | |
| CoTModel=Gemma-42026.06 | 93.25 | |
| TIMETOMModel=Gemini-32026.06 | 92.5 | |
| Ledger promptingBackbone=Llama-3.1-8B2026.05 | 92.17 | |
| Direct promptingBackbone=GPT-OSS-120B2026.05 | 92 | |
| Paradox promptingBackbone=GPT-OSS-120B2026.05 | 91.83 | |
| TIMETOMModel=Qwen3.52026.06 | 91 | |
| Mistral-NeMo-12BParams=12B, Inference Protocol=direct model inference2026.05 | 90.5 | |
| UserHarnessBackbone=GPT-OSS-20B2026.05 | 90.17 | |
| OSCToM-8BParams=8B, Inference Protocol=direct model inference2026.05 | 89.8 | |
| UserHarnessBackbone=Qwen3-8B2026.05 | 89.67 | |
| Direct promptingBackbone=Qwen3-32B2026.05 | 88.5 | |
| SIMTOMModel=Gemma-42026.06 | 88 | |
| Paradox promptingBackbone=GPT-OSS-20B2026.05 | 87.83 | |
| Paradox promptingBackbone=Qwen3-32B2026.05 | 87.33 | |
| Direct promptingBackbone=GPT-5.42026.05 | 86.92 | |
| AutoToMBackbone=GPT-4o2026.05 | 86.92 | |
| Direct promptingBackbone=GPT-OSS-20B2026.05 | 86.75 | |
| UserHarnessBackbone=Gemini-3-Flash2026.05 | 86.25 | |
| Llama-3.1-8B-BaseParams=8B, Inference Protocol=direct model inference2026.05 | 86 | |
| Phi-3-Medium-14BParams=14B, Inference Protocol=direct model inference2026.05 | 85.5 | |
| UserHarnessBackbone=Llama-3.1-8B2026.05 | 82.83 | |
| ExploreToMParams=8B, Inference Protocol=iterative A* heuristic search2026.05 | 81 | |
| TIMETOMModel=Gemma-42026.06 | 80 | |
| Paradox promptingBackbone=GPT-5.4-mini2026.05 | 77.92 | |
| SimToMSetting=reported setting2026.05 | 77.5 | |
| Paradox promptingBackbone=Qwen3-14B2026.05 | 77.08 | |
| Direct promptingBackbone=GPT-5.4-mini2026.05 | 76.42 | |
| Paradox promptingBackbone=Qwen3-8B2026.05 | 76.42 | |
| Direct promptingBackbone=Llama-3.1-8B2026.05 | 74.42 | |
| Paradox promptingBackbone=Llama-3.1-8B2026.05 | 74 | |
| Direct promptingBackbone=Qwen3-14B2026.05 | 73.5 | |
| Direct promptingBackbone=Qwen3-8B2026.05 | 72.33 | |
| LIMPSetting=reported setting2026.05 | 61.67 | |
| BIP-ALMSetting=reported setting2026.05 | 50.33 | |
| Qwen2.5-32BParams=32B, Inference Protocol=direct model inference2026.05 | 47.5 | |
| Qwen2.5-14BParams=14B, Inference Protocol=direct model inference2026.05 | 29.5 | |
| Gemma-2-27BParams=27B, Inference Protocol=direct model inference2026.05 | 18.5 |