Multimodal Question Answering on ManyModalQA (test)
92.5Accuracy (Text)MAMMQA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MAMMQABackbone=OpenAI 4o-mini2025.05 | 92.5 | 96.78 | 78.02 | 89.9 | |
| Human2025.05 | 92 | 89.6 | 94 | 91.6 | |
| MAMMQABackbone=Gemini 1.5-Flash 8B2025.05 | 89.76 | 94.52 | 77.33 | 87.91 | |
| MAMMQABackbone=Qwen 2.5 VL 3B Instruct2025.05 | 88.79 | 94.9 | 72.67 | 86.37 | |
| CapCoTBackbone=OpenAI 4o-mini2025.05 | 87.68 | 94.05 | 68.26 | 84.41 | |
| CoTBackbone=OpenAI 4o-mini, No-context variant=false2025.05 | 87.2 | 94.23 | 57.33 | 81.21 | |
| MAMMQABackbone=Qwen 2.5 VL 7B Instruct2025.05 | 87.11 | 96.31 | 77.56 | 87.61 | |
| ToTBackbone=Gemini 1.5-Flash 8B2025.05 | 86.08 | 86.81 | 62.81 | 79.8 | |
| CoTBackbone=Gemini 1.5-Flash 8B, No-context variant=false2025.05 | 86.05 | 91.52 | 68.77 | 82.81 | |
| CapCoTBackbone=Gemini 1.5-Flash 8B2025.05 | 85.74 | 91.4 | 63.14 | 81.34 | |
| ToTBackbone=OpenAI 4o-mini2025.05 | 84.94 | 93.19 | 72.9 | 84.7 | |
| CapCoTBackbone=Qwen 2.5 VL 7B Instruct2025.05 | 83.5 | 92.86 | 71.07 | 83.41 | |
| ToTBackbone=Qwen 2.5 VL 3B Instruct2025.05 | 82.66 | 86.14 | 68.11 | 80.42 | |
| ToTBackbone=Qwen 2.5 VL 7B Instruct2025.05 | 81.95 | 90.41 | 69.29 | 81.89 | |
| CapCoTBackbone=Qwen 2.5 VL 3B Instruct2025.05 | 80.79 | 91.38 | 67.13 | 80.63 | |
| CoTBackbone=Qwen 2.5 VL 3B Instruct, No-context variant=false2025.05 | 70.08 | 75.61 | 50.7 | 66.54 | |
| CoTBackbone=OpenAI 4o-mini, No-context variant=true2025.05 | 68.22 | 70.51 | 59.42 | 66.54 | |
| CoTBackbone=Qwen 2.5 VL 7B Instruct, No-context variant=true2025.05 | 61.8 | 66.73 | 54.53 | 61.46 | |
| CoTBackbone=Qwen 2.5 VL 7B Instruct, No-context variant=false2025.05 | 59.84 | 68.71 | 45.47 | 58.87 | |
| MMQAOracle setting=true2025.05 | 59.3 | 46.3 | 29 | 46.3 | |
| CoTBackbone=Qwen 2.5 VL 3B Instruct, No-context variant=true2025.05 | 58.77 | 64.55 | 59.51 | 58.77 | |
| CoTBackbone=Gemini 1.5-Flash 8B, No-context variant=true2025.05 | 54.93 | 61.15 | 34.77 | 51.41 | |
| UniMMQABackbone=T5 Model, Model Size=3B2025.05 | 49.8 | 58.3 | 40.9 | 52.1 | |
| MMQAOracle setting=false2025.05 | 48.6 | 40.4 | 27.2 | 39.7 | |
| UniMMQABackbone=T5 Model, Model Size=Large2025.05 | 48.5 | 67.5 | 34.9 | 50 | |
| UniMMQABackbone=T5 Model, Model Size=Base2025.05 | 46.6 | 60.7 | 30.2 | 45.4 | |
| Voting2025.05 | 23.7 | 22.9 | 15.5 | 21.1 |