Visual Question Answering on OK-VQA (val)
66.8AccuracyPaLI-X-VPD
Evaluation Results
| Method | Links | |
|---|---|---|
| PaLI-X-VPDScale=55B, Type=specialist2023.12 | 66.8 | |
| PaLM-E-562BEvaluation Protocol=Generalist (one model)2023.03 | 66.1 | |
| PaLI-XScale=55B2023.12 | 66.1 | |
| CogVLMScale=17B2023.12 | 64.7 | |
| PaLI-X-VPDScale=55B, Type=generalist2023.12 | 64.6 | |
| PaLIEvaluation Protocol=Task-specific finetuned2023.03 | 64.5 | |
| PaLI-17BVocabulary setting=Open-vocabulary generation2022.09 | 64.5 | |
| PaLIScale=17B2023.12 | 64.5 | |
| PaLM-E-84BEvaluation Protocol=Task-specific finetuned2023.03 | 63.3 | |
| PaLM-E-66BEvaluation Protocol=Task-specific finetuned2023.03 | 62.9 | |
| InstructBLIP (Vicuna-7B)Backbone=Vicuna-7B2023.12 | 62.1 | |
| PromptCap + GPT-3Evaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Caption, Knowledge Source=GPT-3 (175B)2022.11 | 60.4 | |
| PaLI-3-VPDScale=5B, Type=specialist2023.12 | 60.3 | |
| PaLM-E-12BEvaluation Protocol=Task-specific finetuned2023.03 | 60.1 | |
| PaLI-3Scale=5B2023.12 | 60.1 | |
| REVIVE (Ensemble)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.11 | 58 | |
| FlamingoFew-shot setting=32-shot, Evaluation Protocol=not finetuned2023.03 | 57.8 | |
| FlamingoParameters=80B, Vocabulary setting=Open-vocabulary generation, Shot setting=32-shot2022.09 | 57.8 | |
| Flamingo (80B)Evaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Feature, Knowledge Source=Chinchilla (70B), shots=322022.11 | 57.8 | |
| PaLI-3-VPDScale=5B, Type=generalist2023.12 | 57.5 | |
| REVIVE (Single)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.11 | 56.6 | |
| PaLI-15BVocabulary setting=Open-vocabulary generation2022.09 | 56.5 | |
| PaLM-E-12BEvaluation Protocol=Generalist (one model)2023.03 | 55.5 | |
| KATVocabulary setting=Open-vocabulary generation2022.09 | 54.4 | |
| KAT (Single)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Tags + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.11 | 54.4 | |
| KAT (Ensemble)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Tags + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.11 | 54.4 | |
| PaLI-3BVocabulary setting=Open-vocabulary generation2022.09 | 52.4 | |
| PaLM-E-12BEvaluation Protocol=Generalist (one model), LLM status=frozen2023.03 | 51.5 | |
| Flamingo (80B)Evaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Feature, Knowledge Source=Chinchilla (70B), shots=02022.11 | 50.6 | |
| TRIGEvaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Tags + OCR, Knowledge Source=Wikipedia2022.11 | 50.5 | |
| PICa-FullEvaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Caption + Tags, Knowledge Source=GPT-3 (175B)2022.11 | 48 | |
| BLIP-2 VIT-G FlanT5XXLEvaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Feature, Knowledge Source=FlanT5-XXL (11B), shots=02022.11 | 45.9 | |
| PICa-BaseEvaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Caption + Tags, Knowledge Source=GPT-3 (175B)2022.11 | 43.3 | |
| MAVEXEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Wikipedia + ConceptNet + Google Images2022.11 | 39.4 | |
| Vis-DPREvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Google Search2022.11 | 39.2 | |
| KRISPEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Wikipedia + ConceptNet2022.11 | 38.4 | |
| VLISV (uses VLM)=true, L (uses unimodal language model)=true2023.10 | 34.2 | |
| ConceptBERTEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=ConceptNet2022.11 | 33.7 | |
| BLIP-2V (uses VLM)=true, L (uses unimodal language model)=false2023.10 | 31.7 | |
| BAN + KG + AUGEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Wikipedia + ConceptNet2022.11 | 26.7 | |
| Naïve EnsembleV (uses VLM)=true, L (uses unimodal language model)=true2023.10 | 26.6 | |
| MUTANEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature2022.11 | 26.4 | |
| OPT-IMLV (uses VLM)=false, L (uses unimodal language model)=true2023.10 | 19.1 | |
| FewVLMV (uses VLM)=true, L (uses unimodal language model)=false2023.10 | 16.5 | |
| Question onlyEvaluation Protocol=End-to-End Finetuning2022.11 | 14.9 | |
| VLKDV (uses VLM)=true, L (uses unimodal language model)=false2023.10 | 13.3 | |
| FrozenV (uses VLM)=true, L (uses unimodal language model)=false2023.10 | 5.9 |