Visual Question Answering on VQA v2 (Accuracy and Relative Performance)
81.8AccuracyVanilla 13B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Vanilla 13BModel Scale=13B, Visual Tokens=2880, Multimodal Projector Fine-tuning=false2024.12 | 81.8 | 100 | |
| VisionZip ‡Model Scale=13B, Visual Tokens=640, Multimodal Projector Fine-tuning=true2024.12 | 81.2 | 99.3 | |
| Vanilla 7BBase Model=mini-Gemini 7B, Number of Tokens=576, Protocol=Upper Bound2024.12 | 80.4 | 100 | |
| VanillaRetained Tokens=2880, Fine-tuning=false2024.12 | 80.1 | 100 | |
| Vanilla 7BModel Scale=7B, Visual Tokens=2880, Multimodal Projector Fine-tuning=false2024.12 | 80.1 | 97.9 | |
| VisionZip ‡Model Scale=13B, Visual Tokens=320, Multimodal Projector Fine-tuning=true2024.12 | 80 | 97.8 | |
| VisionZip ‡Retained Tokens=640, Fine-tuning=true2024.12 | 79.9 | 99.8 | |
| VisionZipModel Scale=13B, Visual Tokens=640, Multimodal Projector Fine-tuning=false2024.12 | 79.7 | 96.9 | |
| VisionZipRetained Tokens=640, Fine-tuning=false2024.12 | 79.1 | 98.8 | |
| VisionZipBase Model=mini-Gemini 7B, Number of Tokens=1922024.12 | 79.1 | 98.4 | |
| VisionZipBase Model=mini-Gemini 7B, Number of Tokens=192, Fine-tuning Protocol=Fine-tuned multimodal projector2024.12 | 78.9 | 98.1 | |
| VisionZip ‡Retained Tokens=320, Fine-tuning=true2024.12 | 78.4 | 97.9 | |
| VisionZipBase Model=mini-Gemini 7B, Number of Tokens=128, Fine-tuning Protocol=Fine-tuned multimodal projector2024.12 | 78.3 | 97.4 | |
| VisionZip ‡Model Scale=13B, Visual Tokens=160, Multimodal Projector Fine-tuning=true2024.12 | 77.6 | 94.9 | |
| VisionZipBase Model=mini-Gemini 7B, Number of Tokens=1282024.12 | 77.5 | 96.4 | |
| VisionZipModel Scale=13B, Visual Tokens=320, Multimodal Projector Fine-tuning=false2024.12 | 76.8 | 93.9 | |
| VisionZipRetained Tokens=320, Fine-tuning=false2024.12 | 76.2 | 95.1 | |
| VisionZipBase Model=mini-Gemini 7B, Number of Tokens=64, Fine-tuning Protocol=Fine-tuned multimodal projector2024.12 | 75.9 | 94.4 | |
| VisionZip ‡Retained Tokens=160, Fine-tuning=true2024.12 | 75.6 | 94.4 | |
| VisionZipBase Model=mini-Gemini 7B, Number of Tokens=642024.12 | 73.9 | 91.9 | |
| VisionZipModel Scale=13B, Visual Tokens=160, Multimodal Projector Fine-tuning=false2024.12 | 72.4 | 88.5 | |
| VisionZipRetained Tokens=160, Fine-tuning=false2024.12 | 71.4 | 89.1 |