Few-Shot Visual Question Answering on VQA V2
53.31Accuracy (1-Shot)Lever-LM
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Lever-LMBackbone=IDEFICS2023.12 | 53.31 | 53.98 | 53.65 | 54.39 | 54.58 | 55.09 | 55.3 | 54.84 | 54.44 | — | |
| SIIRBackbone=IDEFICS2023.12 | 52.87 | 53.28 | 53.08 | 53.77 | 53.92 | 54.81 | 54.88 | 54.35 | 53.92 | — | |
| RSBackbone=IDEFICS2023.12 | 52.4 | 53.21 | 52.81 | 53.47 | 53.7 | 54 | 54.48 | 53.91 | 53.54 | — | |
| SITRBackbone=IDEFICS2023.12 | 51.52 | 51.72 | 51.62 | 52.55 | 52.59 | 52.83 | 50.49 | 52.12 | 51.95 | — | |
| STTRBackbone=IDEFICS2023.12 | 48.13 | 49.32 | 48.73 | 49.43 | 50.11 | 50.58 | 50.93 | 50.26 | 49.75 | — | |
| Lever-LMBackbone=Open-Flamingo2023.12 | 46.66 | 50.83 | 48.75 | 51.91 | 52.15 | 53.29 | 53.01 | 52.59 | 51.31 | — | |
| STTRBackbone=Open-Flamingo2023.12 | 44.6 | 46.75 | 45.68 | 47.92 | 49.05 | 50.06 | 49.47 | 49.13 | 47.98 | — | |
| SIIRBackbone=Open-Flamingo2023.12 | 43.31 | 47.46 | 45.39 | 49.85 | 50.68 | 53.23 | 53.58 | 51.84 | 49.69 | — | |
| LiMBeR + AAPEEvaluation Protocol=train/finetune2024.10 | 42.7 | 44.2 | — | — | 45.9 | — | — | — | — | 36.5 | |
| RSBackbone=Open-Flamingo2023.12 | 41.97 | 45.92 | 43.95 | 48.17 | 48.95 | 51.18 | 51.44 | 49.94 | 47.94 | — | |
| LiMBeR + AAPEEvaluation Protocol=zero-shot2024.10 | 41 | 42.3 | — | — | 43.1 | — | — | — | — | 34.9 | |
| SITRBackbone=Open-Flamingo2023.12 | 40.17 | 43.58 | 41.88 | 46.03 | 47.5 | 49.72 | 50.75 | 48.5 | 46.29 | — | |
| LiMBeR2024.10 | 39.9 | 40.8 | — | — | 40.3 | — | — | — | — | 33.3 | |
| MAGMA2024.10 | 39.3 | 40.6 | — | — | 41.5 | — | — | — | — | 24.6 |