Theory of Mind reasoning on MMToM-QA
98.5Overall AccuracyUserHarness
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| UserHarnessBackbone=Claude-Sonnet-4.62026.05 | 98.5 | — | — | |
| UserHarnessBackbone=Claude-Opus-4.72026.05 | 98.33 | — | — | |
| UserHarnessBackbone=Qwen3-14B2026.05 | 98.17 | — | — | |
| UserHarnessBackbone=GPT-5.4-mini2026.05 | 98.17 | — | — | |
| UserHarnessBackbone=Qwen3-32B2026.05 | 98 | — | — | |
| UserHarnessBackbone=GPT-5.42026.05 | 97.83 | — | — | |
| UserHarnessBackbone=Gemini-3.1-Pro2026.05 | 97.83 | — | — | |
| UserHarnessBackbone=Qwen3-8B2026.05 | 97.67 | — | — | |
| UserHarnessBackbone=Gemini-3-Flash2026.05 | 97.5 | — | — | |
| UserHarnessBackbone=GPT-OSS-120B2026.05 | 97.33 | — | — | |
| UserHarnessBackbone=GPT-OSS-20B2026.05 | 97 | — | — | |
| UserHarnessBackbone=Llama-3.1-8B2026.05 | 96.67 | — | — | |
| AutoToMBackbone=GPT-4o2026.05 | 83 | — | — | |
| Ledger promptingBackbone=GPT-OSS-120B2026.05 | 69.83 | — | — | |
| Paradox promptingBackbone=GPT-OSS-120B2026.05 | 67.5 | — | — | |
| VisionToM w/ Qwen2.5-VL-7B-InstructSetting=Video Only, Nframe=82026.03 | 66.3 | 70.7 | 62 | |
| Direct promptingBackbone=GPT-OSS-120B2026.05 | 64.17 | — | — | |
| Paradox promptingBackbone=GPT-5.42026.05 | 63.33 | — | — | |
| Ledger promptingBackbone=GPT-5.42026.05 | 62.83 | — | — | |
| BIP-ALM-LLaMA2Setting=Video Only, Nframe=-2026.03 | 61.2 | 64 | 58.3 | |
| Qwen2.5-VL-7B-Instruct TransferSetting=Video Only, Nframe=8, Transfer=true2026.03 | 60.5 | 64.3 | 56.7 | |
| BIP-ALMSetting=reported setting2026.05 | 56.17 | — | — | |
| LIMPSetting=reported setting2026.05 | 55.33 | — | — | |
| SimToMSetting=reported setting2026.05 | 51 | — | — | |
| Ledger promptingBackbone=GPT-OSS-20B2026.05 | 48.83 | — | — | |
| Direct promptingBackbone=GPT-5.42026.05 | 48.67 | — | — | |
| Paradox promptingBackbone=Qwen3-32B2026.05 | 48 | — | — | |
| Direct promptingBackbone=GPT-OSS-20B2026.05 | 47.33 | — | — | |
| Ledger promptingBackbone=Llama-3.1-8B2026.05 | 47.33 | — | — | |
| Paradox promptingBackbone=Llama-3.1-8B2026.05 | 47 | — | — | |
| Ledger promptingBackbone=Qwen3-32B2026.05 | 47 | — | — | |
| Paradox promptingBackbone=GPT-OSS-20B2026.05 | 46.67 | — | — | |
| Ledger promptingBackbone=Qwen3-8B2026.05 | 45.67 | — | — | |
| Paradox promptingBackbone=Qwen3-14B2026.05 | 45.17 | — | — | |
| Ledger promptingBackbone=Qwen3-14B2026.05 | 44.33 | — | — | |
| Paradox promptingBackbone=GPT-5.4-mini2026.05 | 43.5 | — | — | |
| Direct promptingBackbone=Llama-3.1-8B2026.05 | 43.33 | — | — | |
| Paradox promptingBackbone=Qwen3-8B2026.05 | 39.67 | — | — | |
| Qwen2.5-VL-7B-InstructSetting=Video Only, Nframe=82026.03 | 38.2 | 47 | 29.3 | |
| Direct promptingBackbone=Qwen3-32B2026.05 | 37.5 | — | — | |
| Direct promptingBackbone=Qwen3-8B2026.05 | 37.17 | — | — | |
| Direct promptingBackbone=Qwen3-14B2026.05 | 37.17 | — | — | |
| Direct promptingBackbone=GPT-5.4-mini2026.05 | 35 | — | — | |
| Ledger promptingBackbone=GPT-5.4-mini2026.05 | 34 | — | — |