Multimodal Vision-Language Evaluation on MMB, MME, GQA, POPE, and SQA^I
101.09MMB ScoreGRACE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| GRACEPre.=W4A16, Token=576, GFLOPs=100.00%2026.04 | 101.09 | — | 98.71 | 99.19 | 97.91 | 99.23 | |
| OriginalPre.=FP16, Token=576, GFLOPs=100.00%2026.04 | 100 | 100 | 100 | 100 | 100 | 100 | |
| VTWPre.=FP16, Token=236, GFLOPs=47.32%2026.04 | 98.6 | 97.64 | 82.74 | 95.69 | 99.14 | 93.8 | |
| DivPrunePre.=FP16, Token=128, GFLOPs=21.73%2026.04 | 96.58 | 93.03 | 94.84 | 99.18 | 98.56 | 96.44 | |
| PDropPre.=FP16, Token=192, GFLOPs=34.03%2026.04 | 95.8 | 92.6 | 92.1 | 97.44 | 98.56 | 95.3 | |
| FastVPre.=FP16, Token=172, GFLOPs=43.20%2026.04 | 95.8 | 96.97 | 92.9 | 94.41 | 99.28 | 95.87 | |
| AWQPre.=W4A16, Token=576, GFLOPs=100.00%2026.04 | 94.32 | 97.04 | 99.19 | 92.19 | — | 95.69 | |
| QUOTAPre.=W4A4KV4, Token=173, GFLOPs=40.89%2026.04 | 92.14 | 94.81 | 95.9 | 98.77 | 95.26 | 95.38 | |
| GPTQPre.=W4A16, Token=576, GFLOPs=100.00%2026.04 | 91.8 | 96.61 | 99.03 | 91.35 | — | 94.59 | |
| LUQPre.=W2.54A16, Token=576, GFLOPs=100.00%2026.04 | 73.66 | 87.2 | 93.87 | 89.54 | — | 86.07 |