Murder Mystery Game Evaluation on Murder Mystery Games
58.75MMRGPT-4V
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| GPT-4VModel Category=Proprietary VLMs2026.04 | 58.75 | 26.43 | 6.43 | 24.2 | 92.4 | 51.88 | 69.25 | |
| Ours (Full Model)Backbone=Qwen2.5-VL-7B-Instruct2026.04 | 58.42 | 36.18 | 6.82 | 36.87 | 89.15 | 77.28 | 62.53 | |
| ClaudeModel Category=Proprietary VLMs2026.04 | 57.78 | 22.07 | 7.89 | 19.2 | 88.8 | 35.31 | 55.02 | |
| Gemini-1.5-ProModel Category=Proprietary VLMs2026.04 | 57.39 | 19.2 | 7.22 | 16.9 | 88.8 | 57.78 | 57.84 | |
| Ours (Full Model)Backbone=Qwen2.5-VL-3B-Instruct2026.04 | 55.01 | 34.25 | 6.35 | 35 | 87.4 | 74.56 | 61.09 | |
| w/o Image Clues MatchBackbone=Qwen2.5-VL-7B-Instruct2026.04 | 50.22 | 34.91 | 5.44 | 32.54 | 78.66 | 72.48 | 45.23 | |
| w/o Supervised Fine-tuningBackbone=Qwen2.5-VL-7B-Instruct2026.04 | 50.12 | 29.36 | 5.68 | 35.48 | 83.27 | 71.02 | 61.42 | |
| Gemma3-12B-itModel Category=Open-source VLMs2026.04 | 49.96 | 33.22 | 7.34 | 19.5 | 81.25 | 65.43 | 55.01 | |
| w/o Image Clues MatchBackbone=Qwen2.5-VL-3B-Instruct2026.04 | 48.75 | 33.25 | 5.15 | 31.25 | 77.13 | 71.01 | 44.05 | |
| w/o Supervised Fine-tuningBackbone=Qwen2.5-VL-3B-Instruct2026.04 | 48.56 | 27.84 | 5.32 | 34.32 | 82.04 | 69.76 | 60.1 | |
| Gemma3-27B-itModel Category=Open-source VLMs2026.04 | 48.28 | 26.91 | 7.61 | 15.83 | 76.34 | 64.24 | 56.42 | |
| w/o Reinforcement LearningBackbone=Qwen2.5-VL-7B-Instruct2026.04 | 47.4 | 18.5 | 5.12 | 26.13 | 86.41 | 63.85 | 59.73 | |
| w/o Reinforcement LearningBackbone=Qwen2.5-VL-3B-Instruct2026.04 | 45.83 | 17.02 | 4.76 | 24.62 | 85.27 | 61.97 | 58.88 | |
| Mistral-small3.1-24BModel Category=Open-source VLMs2026.04 | 44.92 | 40.43 | 7.53 | 33.09 | 83.07 | 50.74 | 48.41 | |
| Gemma3-4B-itModel Category=Open-source VLMs2026.04 | 40.53 | 18.21 | 7.16 | 17.09 | 34.7 | 23.5 | 22.38 | |
| Qwen2.5-VL-7B-Instruct (baseline)Model Category=Open-source VLMs2026.04 | 37.63 | 30.7 | 7.11 | 25.61 | 83.97 | 40.63 | 38.74 | |
| LLaMA3.2-Vision-11BModel Category=Open-source VLMs2026.04 | 33.71 | 26.39 | 3.95 | 11.69 | 57.2 | 24.13 | 22.6 | |
| Qwen2.5-VL-3B-Instruct (baseline)Model Category=Open-source VLMs2026.04 | 30.92 | 23.93 | 4.69 | 20.14 | 72.88 | 35.09 | 32.16 | |
| LLaVA-13BModel Category=Open-source VLMs2026.04 | 19.01 | 20.78 | 2.17 | 15.83 | 23.92 | 21.35 | 18.7 |