Visual Question Answering on VizWiz (test-dev)
76.4AccuracyCogVLM (Specialist)
Evaluation Results
| Method | Links | |
|---|---|---|
| CogVLM (Specialist)OCR pipeline input=false, Model Category=Specialist, Reference Citation=[58]2023.12 | 76.4 | |
| CogVLM (Specialist)OCR pipeline input=true, Model Category=Specialist, Reference Citation=[58]2023.12 | 76.4 | |
| SOTA [58]2023.12 | 76.4 | |
| SMoLA-PaLI-X_FT (Specialist)Base model=PaLI-X, Per-task LoRA tuning=true, LoRA rank=42023.12 | 75.9 | |
| PaLI-XBase model=PaLI-X2023.12 | 74.6 | |
| PaLI-17BVocabulary setting=Open-vocabulary generation2022.09 | 74.4 | |
| PaLI-17Btrained_on_QA_datasets=true2024.03 | 74.4 | |
| SMoLA-PaLI-XFTOCR pipeline input=true, Model Category=Generalist, Backbone=PaLI-X2023.12 | 73.8 | |
| SMoLA-PaLI-XFTOCR pipeline input=false, Model Category=Generalist, Backbone=PaLI-X2023.12 | 71.7 | |
| PaLI-15BVocabulary setting=Open-vocabulary generation2022.09 | 71.1 | |
| SMoLA-PaLI-3FTOCR pipeline input=true, Model Category=Generalist, Backbone=PaLI-32023.12 | 71 | |
| GIT2Parameters=5.1B, Vocabulary setting=Closed-vocabulary classification2022.09 | 70.97 | |
| GIT2Parameters=5.1B2022.05 | 70.97 | |
| SMoLA-PaLI-3FTOCR pipeline input=false, Model Category=Generalist, Backbone=PaLI-32023.12 | 70.4 | |
| GITParameters=0.7B, Vocabulary setting=Closed-vocabulary classification2022.09 | 68 | |
| GITParameters=0.7B2022.05 | 68 | |
| PaLI-3BVocabulary setting=Open-vocabulary generation2022.09 | 67.5 | |
| PaLI-3Btrained_on_QA_datasets=true2024.03 | 67.5 | |
| Flamingomode=Fine-tuned2022.04 | 65.7 | |
| FlamingoEvaluation Protocol=Fine-tuned2022.04 | 65.7 | |
| FlamingoParameters=80B, Vocabulary setting=Open-vocabulary generation2022.09 | 65.7 | |
| Flamingo FTfine-tuned=true, trained_on_QA_datasets=true2024.03 | 65.7 | |
| Flamingo2022.05 | 65.7 | |
| GIT_LParameters=0.3B2022.05 | 62.5 | |
| Liu et al. (2021)##2022.05 | 61.8 | |
| InternVL-ChatLLM=Vicuna-13B, glue layer=QLLAMA2023.12 | 58.5 | |
| Unified-IOOCR pipeline input=false, Model Category=Generalist2023.12 | 57.4 | |
| SotA2022.04 | 57.2 | |
| Unrestricted SotA (Lin et al.)Evaluation Protocol=Fine-tuned2022.04 | 57.2 | |
| InternVL-ChatLLM=Vicuna-13B, glue layer=MLP2023.12 | 54.6 | |
| GIT_BParameters=0.1B2022.05 | 54.6 | |
| LOVA³-7BLLM=Vicuna-7B2024.05 | 54.2 | |
| LLaVA-1.5LLM=Vicuna-13B2023.12 | 53.6 | |
| FasterVLMReduction Ratio=90%, Vision Tokens=582026.02 | 53 | |
| M3Reduction Ratio=75%, Vision Tokens=362026.02 | 52.8 | |
| Mask-LLaVAReduction Ratio=97%, Vision Tokens=152026.02 | 52.8 | |
| FasterVLMReduction Ratio=95%, Vision Tokens=292026.02 | 52.6 | |
| InternVL-ChatLLM=Vicuna-7B, glue layer=MLP2023.12 | 52.5 | |
| Mask-LLaVAReduction Ratio=97%, Vision Tokens=422026.02 | 52.1 | |
| M3Reduction Ratio=75%, Vision Tokens=92026.02 | 51.9 | |
| Mask-LLaVAReduction Ratio=90%, Vision Tokens=572026.02 | 51.8 | |
| MQTReduction Ratio=88%, Vision Tokens=642026.02 | 51.5 | |
| MQTReduction Ratio=88%, Vision Tokens=362026.02 | 51 | |
| FitPruneReduction Ratio=90%, Vision Tokens=582026.02 | 50.8 | |
| InstructBLIPLLM=Vicuna-7B2023.12 | 50.1 | |
| LLaVA-1.5LLM=Vicuna-7B2023.12 | 50 | |
| LLaVA-1.5LLM=Vicuna-7B2024.05 | 50 | |
| LLaVA-1.5-7BReduction Ratio=0%, Vision Tokens=5762026.02 | 50 | |
| FlamingoFew-shot=32 shots2022.04 | 49.8 | |
| Flamingo 32 Shotshots=32, trained_on_QA_datasets=true2024.03 | 49.8 | |
| MQTReduction Ratio=88%, Vision Tokens=162026.02 | 49.8 | |
| SparseVLMReduction Ratio=90%, Vision Tokens=582026.02 | 49.3 | |
| FitPruneReduction Ratio=95%, Vision Tokens=292026.02 | 48.6 | |
| FlexCap-LLMzero-shot=true2024.03 | 41.8 | |
| Qwen-VL-ChatLLM=Qwen-7B2023.12 | 38.9 | |
| SFT-CM3Leon-7Bshots=0-shot, SFT=true2023.09 | 37.6 | |
| LOVA³-1.5BLLM=Phi-1.5B2024.05 | 36.9 | |
| IDEFICS-80BLLM=LLaMA-65B2023.12 | 36 | |
| InstructBLIPzero-shot=true2024.03 | 34.5 | |
| Emuzero-shot=true2024.03 | 34.2 | |
| LLaVA-Phi-1.5LLM=Phi-1.5B2024.05 | 33.8 | |
| Flamingozero-shot=true2024.03 | 31.6 | |
| Flamingo-9Bshots=0-shot2023.09 | 28.8 | |
| IDEFICS-80B-ILLM=LLaMA-65B2023.12 | 26 | |
| BLIP-2LLM=Vicuna-13B2023.12 | 19.6 |