Theory of Mind on ToMi
100AccuracyUserHarness
Evaluation Results
| Method | Links | |
|---|---|---|
| UserHarnessBackbone=Qwen3-8B2026.05 | 100 | |
| UserHarnessBackbone=Qwen3-14B2026.05 | 100 | |
| UserHarnessBackbone=Qwen3-32B2026.05 | 100 | |
| UserHarnessBackbone=Llama-3.1-8B2026.05 | 100 | |
| UserHarnessBackbone=GPT-OSS-20B2026.05 | 100 | |
| UserHarnessBackbone=GPT-OSS-120B2026.05 | 100 | |
| UserHarnessBackbone=GPT-5.4-mini2026.05 | 100 | |
| UserHarnessBackbone=GPT-5.42026.05 | 100 | |
| UserHarnessBackbone=Gemini-3-Flash2026.05 | 100 | |
| UserHarnessBackbone=Gemini-3.1-Pro2026.05 | 100 | |
| UserHarnessBackbone=Claude-Sonnet-4.62026.05 | 100 | |
| UserHarnessBackbone=Claude-Opus-4.72026.05 | 100 | |
| SymbolicToMSetting=reported setting2026.05 | 98.6 | |
| ExploreToMParams=8B, Inference Protocol=iterative A* heuristic search2026.05 | 95 | |
| DITTOBackbone=Qwen3-VL-8B-Instruct2026.05 | 93 | |
| Paradox promptingBackbone=GPT-5.42026.05 | 89.1 | |
| AutoToMBackbone=GPT-4o2026.05 | 88.3 | |
| GPT-5.42026.05 | 88 | |
| Ledger promptingBackbone=GPT-5.42026.05 | 85.72 | |
| GPT-5-nano2026.05 | 85 | |
| Paradox promptingBackbone=GPT-5.4-mini2026.05 | 83.97 | |
| GRPOBackbone=Qwen3-VL-8B-Instruct2026.05 | 82 | |
| Direct promptingBackbone=GPT-OSS-120B2026.05 | 80.22 | |
| SimToMSetting=reported setting2026.05 | 79.9 | |
| Direct promptingBackbone=GPT-5.42026.05 | 79.72 | |
| OSCToM-8BParams=8B, Inference Protocol=direct model inference2026.05 | 79.5 | |
| Direct promptingBackbone=GPT-5.4-mini2026.05 | 78.36 | |
| Ledger promptingBackbone=GPT-OSS-120B2026.05 | 77.38 | |
| Phi-3-Medium-14BParams=14B, Inference Protocol=direct model inference2026.05 | 76 | |
| Paradox promptingBackbone=GPT-OSS-120B2026.05 | 75.05 | |
| Qwen2.5-32BParams=32B, Inference Protocol=direct model inference2026.05 | 74.7 | |
| Ledger promptingBackbone=GPT-5.4-mini2026.05 | 73.03 | |
| HumanLM-8B2026.05 | 72 | |
| Direct promptingBackbone=GPT-OSS-20B2026.05 | 70.08 | |
| Paradox promptingBackbone=GPT-OSS-20B2026.05 | 69.56 | |
| Qwen3-VL-8B-InstructRole=Base2026.05 | 68 | |
| Mistral-NeMo-12BParams=12B, Inference Protocol=direct model inference2026.05 | 66.8 | |
| Llama-3.1-8B-BaseParams=8B, Inference Protocol=direct model inference2026.05 | 64.3 | |
| Ledger promptingBackbone=GPT-OSS-20B2026.05 | 62.37 | |
| BIP-ALMSetting=reported setting2026.05 | 55.6 | |
| Direct promptingBackbone=Qwen3-8B2026.05 | 54.52 | |
| Qwen2.5-14BParams=14B, Inference Protocol=direct model inference2026.05 | 53.8 | |
| Gemma-2-27BParams=27B, Inference Protocol=direct model inference2026.05 | 53.3 | |
| Direct promptingBackbone=Llama-3.1-8B2026.05 | 52.48 | |
| Paradox promptingBackbone=Qwen3-32B2026.05 | 50.38 | |
| Paradox promptingBackbone=Qwen3-14B2026.05 | 48.96 | |
| Ledger promptingBackbone=Qwen3-32B2026.05 | 47.34 | |
| LIMPSetting=reported setting2026.05 | 44.6 | |
| Ledger promptingBackbone=Qwen3-8B2026.05 | 26.44 | |
| Ledger promptingBackbone=Qwen3-14B2026.05 | 25.28 | |
| Paradox promptingBackbone=Qwen3-8B2026.05 | 17.32 | |
| Direct promptingBackbone=Qwen3-32B2026.05 | 14.02 | |
| Ledger promptingBackbone=Llama-3.1-8B2026.05 | 11.24 | |
| Direct promptingBackbone=Qwen3-14B2026.05 | 6.2 | |
| Paradox promptingBackbone=Llama-3.1-8B2026.05 | 5.5 |