Theory of Mind reasoning on MMToM-QA Multimodal
95.8Belief Inference 1.1Human
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| HumanModality=Multimodal2024.01 | 95.8 | 96.7 | 100 | 97.5 | 90 | 91.7 | 83.3 | 88.9 | 88.5 | 93 | |
| HumanModality=multimodal2025.06 | 95.8 | 96.7 | 100 | 97.5 | 90 | 91.7 | 83.3 | 88.9 | 88.5 | 93 | |
| GPT-4VModality=Multimodal2024.01 | 94 | 13 | 59 | 55.3 | 56 | 26.7 | 4 | 52 | 34.7 | 44 | |
| GPT-4VModality=multimodal2025.06 | 94 | 13 | 59 | 55.3 | 56 | 26.7 | 4 | 52 | 34.7 | 44 | |
| Scalable Bayesian ToM PlannerModality=multimodal, Backbone=Llama3.1-405B2025.06 | 92.1 | 76 | 93 | 87.1 | 73.4 | 80 | 75.5 | 78.7 | 76.9 | 81.3 | |
| BIP-ALMModality=Multimodal, Backbone=GPT-J2024.01 | 90 | 69 | 86 | 81.7 | 68 | 78.7 | 56 | 73.3 | 69 | 75.3 | |
| BIPALM w/ GPT-J-6BModality=multimodal2025.06 | 90 | 69 | 86 | 81.7 | 68 | 78.7 | 56 | 73.3 | 69 | 75.3 | |
| BIP-ALMModality=Multimodal, Backbone=LLAMA 22024.01 | 88 | 68 | 85 | 80.3 | 62.7 | 77.3 | 72 | 80 | 73.3 | 76.7 | |
| BIPALM w/ Llama2-7BModality=multimodal2025.06 | 88 | 68 | 85 | 80.3 | 62.7 | 77.3 | 72 | 80 | 73.3 | 76.7 | |
| InstructBLIPModality=Multimodal2024.01 | 62 | 52 | 32 | 48.7 | 46.7 | 29.3 | 42.7 | 60 | 44.7 | 46.7 | |
| LLaVAModality=Multimodal2024.01 | 46 | 14 | 69 | 43 | 65.3 | 22.7 | 40 | 48 | 44 | 43.5 | |
| LLaVA-7BModality=multimodal2025.06 | 46 | 14 | 69 | 43 | 65.3 | 22.7 | 40 | 48 | 44 | 43.5 | |
| Video-LLaMA 2Modality=Multimodal2024.01 | 36 | 38 | 52 | 42 | 36 | 41.3 | 30.7 | 45.3 | 38.3 | 40.2 | |
| Video-Llama2-13BModality=multimodal2025.06 | 36 | 38 | 52 | 42 | 36 | 41.3 | 30.7 | 45.3 | 38.3 | 40.2 |