Multimodal Evaluation Suite
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
Multimodal Evaluation Suite Average
100Average Relative Performance
21
Multimodal Evaluation Suite (GQA, MMBench, MMBench-CN, MME, POPE, SEED-Bench, TextVQA, VizWiz, OCRBench)
61.5GQA Score
21
Multimodal Evaluation Suite (Tiny-ImageNet, CIFAR-100, FMNIST, Caltech-256, AG News, MMLU, VQA, CommonGen) (test)
100Significance Rate (TAP Better)
18
Multimodal Evaluation Suite MMB, MME, SQA, VQA^T, MMB^C, MMVet, MMstar, AI2D
64.6MMB Score
17
Multimodal Evaluation Suite GQA, MMBench, MMBench-CN, MME, POPE, ScienceQA, TextVQA (various)
62.2GQA Accuracy
16
Multimodal Evaluation Suite (MME, TextVQA, DocVQA, AI2D, MMMU, MMBench)
1,687.9MME Score
13
Multimodal Evaluation Suite (MMVet, MMBench_EN, SEED-Bench, LLaVABench, POPE, MME-P, MMVP, MMStar) (Random Sampling Splits of CC12M)
30.1MMVet Score
13
Multimodal Evaluation Suite (VQAv2, GQA, SQAImg, VQAText, POPE, MME, MMBEN, MMBCN, MMVet)
81.3VQAv2 Accuracy
10
Multimodal Evaluation Suite Table 4
73.6ALL AVG Score
9
Multimodal Evaluation Suite GQA, SQA-I, VQA-T, MME, VQAv2, MMB
59.7GQA Score
7
Multimodal Evaluation Suite Composite
68.7Overall Score
5