Theory of Mind on HiToM
87.08AccuracyUserHarness
Evaluation results
| Method | Links | |
|---|---|---|
| UserHarnessBackbone=Qwen3-32B2026.05 | 87.08 | |
| UserHarnessBackbone=Qwen3-14B2026.05 | 87 | |
| UserHarnessBackbone=GPT-OSS-20B2026.05 | 86.83 | |
| UserHarnessBackbone=GPT-5.42026.05 | 86.83 | |
| UserHarnessBackbone=Claude-Opus-4.72026.05 | 86.83 | |
| UserHarnessBackbone=GPT-OSS-120B2026.05 | 86.75 | |
| UserHarnessBackbone=Qwen3-8B2026.05 | 86.58 | |
| UserHarnessBackbone=GPT-5.4-mini2026.05 | 86.58 | |
| UserHarnessBackbone=Llama-3.1-8B2026.05 | 86.42 | |
| UserHarnessBackbone=Gemini-3-Flash2026.05 | 86.42 | |
| UserHarnessBackbone=Gemini-3.1-Pro2026.05 | 86.42 | |
| UserHarnessBackbone=Claude-Sonnet-4.62026.05 | 86.25 | |
| DITTOBackbone=Qwen3-VL-8B-Instruct2026.05 | 78 | |
| GRPOBackbone=Qwen3-VL-8B-Instruct2026.05 | 77 | |
| Ledger promptingBackbone=GPT-5.42026.05 | 77 | |
| Paradox promptingBackbone=GPT-5.42026.05 | 75.5 | |
| Ledger promptingBackbone=GPT-OSS-120B2026.05 | 75.08 | |
| Direct promptingBackbone=GPT-OSS-120B2026.05 | 74.92 | |
| GPT-o3Model Family=GPT Family, Reasoning Variant=Reasoning2026.02 | 74.7 | |
| Paradox promptingBackbone=GPT-OSS-120B2026.05 | 73.42 | |
| AutoToMBackbone=GPT-4o2026.05 | 72.5 | |
| Direct promptingBackbone=GPT-5.42026.05 | 72.08 | |
| SimToMSetting=reported setting2026.05 | 71 | |
| GPT-5.42026.05 | 70 | |
| DeepSeek-V3Model Family=DeepSeek Family, Reasoning Variant=Non-Reasoning2026.02 | 69.4 | |
| Qwen2.5-32BParams=32B, Inference Protocol=direct model inference2026.05 | 68.5 | |
| Qwen3-32B-ReasoningModel Family=Qwen3-32B, Reasoning Variant=Reasoning2026.02 | 68 | |
| Llama-3.1-8B-BaseParams=8B, Inference Protocol=direct model inference2026.05 | 67.8 | |
| Ledger promptingBackbone=GPT-OSS-20B2026.05 | 67.5 | |
| Direct promptingBackbone=GPT-OSS-20B2026.05 | 66.83 | |
| Paradox promptingBackbone=GPT-OSS-20B2026.05 | 65.92 | |
| Phi-3-Medium-14BParams=14B, Inference Protocol=direct model inference2026.05 | 65.3 | |
| OSCToM-8BParams=8B, Inference Protocol=direct model inference2026.05 | 65.3 | |
| GPT-4oModel Family=GPT Family, Reasoning Variant=Non-Reasoning2026.02 | 60.7 | |
| Paradox promptingBackbone=GPT-5.4-mini2026.05 | 60.58 | |
| ExploreToMParams=8B, Inference Protocol=iterative A* heuristic search2026.05 | 59 | |
| Ledger promptingBackbone=GPT-5.4-mini2026.05 | 58.92 | |
| Qwen3-32BModel Family=Qwen3-32B, Reasoning Variant=Non-Reasoning2026.02 | 58.6 | |
| Qwen3-VL-8B-InstructRole=Base2026.05 | 58 | |
| Qwen2.5-14BParams=14B, Inference Protocol=direct model inference2026.05 | 57.9 | |
| Direct promptingBackbone=GPT-5.4-mini2026.05 | 57.08 | |
| HER-32B2026.05 | 56 | |
| Direct promptingBackbone=Qwen3-14B2026.05 | 56 | |
| Qwen3-8BModel Family=Qwen3-8B, Reasoning Variant=Non-Reasoning2026.02 | 55.8 | |
| DeepSeek-R1Model Family=DeepSeek Family, Reasoning Variant=Reasoning2026.02 | 54.9 | |
| GPT-o4-miniModel Family=GPT Family, Reasoning Variant=Non-Reasoning2026.02 | 54.7 | |
| Direct promptingBackbone=Qwen3-32B2026.05 | 53.42 | |
| Direct promptingBackbone=Qwen3-8B2026.05 | 50.92 | |
| Qwen3-8B-ReasoningModel Family=Qwen3-8B, Reasoning Variant=Reasoning2026.02 | 48.1 | |
| SymbolicToMSetting=reported setting2026.05 | 44.5 | |
| Direct promptingBackbone=Llama-3.1-8B2026.05 | 41.08 | |
| GPT-5-nano2026.05 | 37 | |
| Ledger promptingBackbone=Qwen3-14B2026.05 | 36 | |
| Mistral-NeMo-12BParams=12B, Inference Protocol=direct model inference2026.05 | 25.6 | |
| Ledger promptingBackbone=Qwen3-32B2026.05 | 24.75 | |
| Gemma-2-27BParams=27B, Inference Protocol=direct model inference2026.05 | 20.6 | |
| BIP-ALMSetting=reported setting2026.05 | 14.5 | |
| Paradox promptingBackbone=Qwen3-14B2026.05 | 12.67 | |
| Ledger promptingBackbone=Qwen3-8B2026.05 | 9.75 | |
| Ledger promptingBackbone=Llama-3.1-8B2026.05 | 8.17 | |
| LIMPSetting=reported setting2026.05 | 6.5 | |
| Paradox promptingBackbone=Qwen3-32B2026.05 | 3.58 | |
| Paradox promptingBackbone=Llama-3.1-8B2026.05 | 1.5 | |
| Paradox promptingBackbone=Qwen3-8B2026.05 | 0 |