Mental State Inference on MMToM-QA human 1.0 (test)
100Sub-score 1.1GPT-4
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4Modality=Text only2024.01 | 100 | 50.9 | 65.3 | 36.7 | 31.5 | 64 | 40 | 40 | 13.3 | 53.3 | |
| SimToMModality=Text only, LLM Backbone=GPT-42024.01 | 100 | 56.7 | 70 | 43.4 | 31.6 | 80 | 46.7 | 40 | 20 | 66.7 | |
| SymbolicToMModality=Text only, LLM Backbone=GPT-42024.01 | 100 | 53.3 | 70 | 36.7 | 78.9 | 44 | 40 | 73.3 | 0 | 33.3 | |
| BIP-ALMModality=Multimodal, LLM Backbone=GPT-J2024.01 | 93.8 | 76.6 | 78.1 | 75 | 52.6 | 88 | 73.3 | 86.7 | 73.3 | 66.7 | |
| GPT-3.5Modality=Text only2024.01 | 93.8 | 45.7 | 58.1 | 33.3 | 52.6 | 28 | 53.3 | 0 | 33.3 | 46.7 | |
| GPT-4VModality=Multimodal2024.01 | 93.4 | 48.9 | 52.7 | 45 | 36.8 | 28 | 33.3 | 73.3 | 13.3 | 60 | |
| BIP-ALMModality=Multimodal, LLM Backbone=LLaMA 22024.01 | 87.5 | 77 | 85.7 | 68.3 | 73.7 | 96 | 60 | 86.7 | 60 | 66.7 | |
| LLaMA 2Modality=Text only2024.01 | 75 | 53.6 | 57.2 | 50 | 52.6 | 44 | 46.7 | 40 | 60 | 53.3 | |
| BIP-ALMModality=Text only, LLM Backbone=GPT-J2024.01 | 75 | 62.6 | 71.9 | 53.3 | 52.6 | 88 | 40 | 73.3 | 33.3 | 66.7 | |
| BIP-ALMModality=Text only, LLM Backbone=LLaMA 22024.01 | 75 | 67.9 | 75.9 | 60 | 52.6 | 100 | 66.7 | 80 | 53.3 | 40 | |
| BIP-ALMModality=Video only, LLM Backbone=LLaMA 22024.01 | 75 | 63.1 | 69.6 | 56.7 | 57.9 | 76 | 60 | 53.3 | 60 | 53.3 | |
| GPT-JModality=Text only2024.01 | 62.5 | 45.6 | 49.5 | 41.7 | 42.1 | 44 | 33.3 | 53.3 | 26.7 | 53.3 | |
| BIP-ALMModality=Video only, LLM Backbone=GPT-J2024.01 | 62.5 | 59.2 | 60.1 | 58.3 | 57.9 | 60 | 46.7 | 53.3 | 73.3 | 60 | |
| GPT-4VModality=Video only2024.01 | 56.3 | 44.8 | 41.2 | 48.3 | 47.4 | 20 | 33.3 | 40 | 60 | 60 | |
| Video-LLaMA 2Modality=Multimodal2024.01 | 50 | 36.9 | 40.5 | 33.3 | 47.4 | 24 | 33.3 | 33.3 | 20 | 46.7 | |
| InstructBLIPModality=Multimodal2024.01 | 43.8 | 49.1 | 49.9 | 48.3 | 57.9 | 48 | 53.3 | 46.7 | 40 | 53.3 | |
| InstructBLIPModality=Video only2024.01 | 43.8 | 49.1 | 48.1 | 50 | 52.6 | 48 | 46.7 | 46.7 | 46.7 | 60 | |
| Video-LLaMA 2Modality=Video only2024.01 | 31.3 | 45.2 | 42 | 48.3 | 26.8 | 68 | 53.3 | 60 | 40 | 40 | |
| LLaVAModality=Multimodal2024.01 | 24 | 50.9 | 60.1 | 41.7 | 56.3 | 100 | 40 | 33.3 | 40 | 53.3 | |
| LLaVAModality=Video only2024.01 | 4 | 39.6 | 40.9 | 38.3 | 18.8 | 100 | 46.7 | 26.7 | 40 | 40 |