List your company
ResearchFormerly WizWand

Visual Question Answering on OK-VQA (val)

66.8AccuracyPaLI-X-VPD
3.46419.90736.3552.793Sep 14, 2022Nov 27, 2022Feb 10, 2023Apr 25, 2023Jul 9, 2023Sep 21, 2023Dec 5, 2023
Updated 8mo ago

Evaluation results

MethodLinks
PaLI-X-VPDScale=55B, Type=specialist2023.1266.8
PaLM-E-562BEvaluation Protocol=Generalist (one model)2023.0366.1
PaLI-XScale=55B2023.1266.1
CogVLMScale=17B2023.1264.7
PaLI-X-VPDScale=55B, Type=generalist2023.1264.6
PaLIEvaluation Protocol=Task-specific finetuned2023.0364.5
PaLI-17BVocabulary setting=Open-vocabulary generation2022.0964.5
PaLIScale=17B2023.1264.5
PaLM-E-84BEvaluation Protocol=Task-specific finetuned2023.0363.3
PaLM-E-66BEvaluation Protocol=Task-specific finetuned2023.0362.9
InstructBLIP (Vicuna-7B)Backbone=Vicuna-7B2023.1262.1
PromptCap + GPT-3Evaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Caption, Knowledge Source=GPT-3 (175B)2022.1160.4
PaLI-3-VPDScale=5B, Type=specialist2023.1260.3
PaLM-E-12BEvaluation Protocol=Task-specific finetuned2023.0360.1
PaLI-3Scale=5B2023.1260.1
REVIVE (Ensemble)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.1158
FlamingoFew-shot setting=32-shot, Evaluation Protocol=not finetuned2023.0357.8
FlamingoParameters=80B, Vocabulary setting=Open-vocabulary generation, Shot setting=32-shot2022.0957.8
Flamingo (80B)Evaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Feature, Knowledge Source=Chinchilla (70B), shots=322022.1157.8
PaLI-3-VPDScale=5B, Type=generalist2023.1257.5
REVIVE (Single)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.1156.6
PaLI-15BVocabulary setting=Open-vocabulary generation2022.0956.5
PaLM-E-12BEvaluation Protocol=Generalist (one model)2023.0355.5
KATVocabulary setting=Open-vocabulary generation2022.0954.4
KAT (Single)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Tags + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.1154.4
KAT (Ensemble)Evaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Tags + Feature, Knowledge Source=GPT-3 (175B) + Wikidata2022.1154.4
PaLI-3BVocabulary setting=Open-vocabulary generation2022.0952.4
PaLM-E-12BEvaluation Protocol=Generalist (one model), LLM status=frozen2023.0351.5
Flamingo (80B)Evaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Feature, Knowledge Source=Chinchilla (70B), shots=02022.1150.6
TRIGEvaluation Protocol=End-to-End Finetuning, Image Representation=Caption + Tags + OCR, Knowledge Source=Wikipedia2022.1150.5
PICa-FullEvaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Caption + Tags, Knowledge Source=GPT-3 (175B)2022.1148
BLIP-2 VIT-G FlanT5XXLEvaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Feature, Knowledge Source=FlanT5-XXL (11B), shots=02022.1145.9
PICa-BaseEvaluation Protocol=In-Context Learning & Zero-Shot, Image Representation=Caption + Tags, Knowledge Source=GPT-3 (175B)2022.1143.3
MAVEXEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Wikipedia + ConceptNet + Google Images2022.1139.4
Vis-DPREvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Google Search2022.1139.2
KRISPEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Wikipedia + ConceptNet2022.1138.4
VLISV (uses VLM)=true, L (uses unimodal language model)=true2023.1034.2
ConceptBERTEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=ConceptNet2022.1133.7
BLIP-2V (uses VLM)=true, L (uses unimodal language model)=false2023.1031.7
BAN + KG + AUGEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature, Knowledge Source=Wikipedia + ConceptNet2022.1126.7
Naïve EnsembleV (uses VLM)=true, L (uses unimodal language model)=true2023.1026.6
MUTANEvaluation Protocol=End-to-End Finetuning, Image Representation=Feature2022.1126.4
OPT-IMLV (uses VLM)=false, L (uses unimodal language model)=true2023.1019.1
FewVLMV (uses VLM)=true, L (uses unimodal language model)=false2023.1016.5
Question onlyEvaluation Protocol=End-to-End Finetuning2022.1114.9
VLKDV (uses VLM)=true, L (uses unimodal language model)=false2023.1013.3
FrozenV (uses VLM)=true, L (uses unimodal language model)=false2023.105.9