Visual Reasoning on NLVR2
92.6AccuracyBEIT-3
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BEIT-32022.08 | 92.6 | — | |
| mPLUG-Owl3# Param=8B2024.08 | 90.8 | — | |
| MantisSOFA=false2025.03 | 90.26 | — | |
| MantisSOFA=true2025.03 | 89.98 | — | |
| LLaVA-NeXTSOFA=true2025.03 | 89.45 | — | |
| LLAVA-Interleave# Param=8B2024.08 | 88.8 | — | |
| InternVL2SOFA=true2025.03 | 88.19 | — | |
| Mantis-SigLIP# Param=8B2024.08 | 87.4 | — | |
| LLaVA-NeXTSOFA=false2025.03 | 87.28 | — | |
| CoCa2022.08 | 87 | — | |
| Idefics2# Param=8B2024.08 | 86.9 | — | |
| InternVL2SOFA=false2025.03 | 85.56 | — | |
| Idefics2SOFA=true2025.03 | 85.15 | — | |
| ITNet-LParams=307M2026.06 | 84.1 | — | |
| BLIP (ViT-L)Params=385M2026.06 | 83.5 | — | |
| ALBEFParams=314M2026.06 | 82.6 | — | |
| BLIP (ViT-B)Params=250M2026.06 | 82.3 | — | |
| ITNet-BParams=86M2026.06 | 82.1 | — | |
| Idefics2SOFA=false2025.03 | 81.94 | — | |
| METER-CLIPParams=185M2026.06 | 81.5 | — | |
| Brote-IM-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 80.71 | — | |
| Brote-IM-XXL (ours)#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 80.71 | — | |
| VILASOFA=true2025.03 | 80.34 | — | |
| UNITER-BParams=110M2026.06 | 79.5 | — | |
| VILASOFA=false2025.03 | 78.82 | — | |
| ITNet-SParams=22M2026.06 | 78.5 | — | |
| IndividualBackbone=BEiT3-base2024.05 | 77.65 | — | |
| VILA# Param=8B2024.08 | 76.5 | — | |
| Brote-IM-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 76.02 | — | |
| Brote-IM-XL (ours)#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 76.02 | — | |
| ViLTParams=86M2026.06 | 75.9 | — | |
| EMR-MERGINGBackbone=BEiT3-base2024.05 | 74.75 | — | |
| MMICL-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 71.48 | — | |
| MMICL-XL#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 71.48 | — | |
| Brote-EX-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 71.41 | — | |
| Brote-EX-XL (ours)#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 71.41 | — | |
| Brote-EX-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 70.42 | — | |
| Brote-EX-XXL (ours)#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 70.42 | — | |
| VisualBERTInteraction=Fine-grained vision-language interaction2022.09 | 67 | — | |
| ERNIE-ViL 2.0Interaction=Shallow cross-modal interaction, Model Type=English pre-trained2022.09 | 66 | — | |
| InstructBLIP-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 64.54 | — | |
| InstructBLIP-XXL#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 64.54 | — | |
| InstructBLIP# Param=8B2024.08 | 60.3 | — | |
| Qwen-VL-Chat# Param=8B2024.08 | 58.7 | — | |
| CogVLM# Param=17B2024.08 | 58.6 | — | |
| VideoLLaVA# Param=8B2024.08 | 56.5 | — | |
| MMICL-XXLLLM Backbone=FlanT5, #Param LLM=11B2024.02 | 56.16 | — | |
| MMICL-XXL#Param LLM=11B, Task Setting=Multi-image / Video Tasks2024.02 | 56.16 | — | |
| VPG-C-LLAMA2LLM Backbone=LLAMA, #Param LLM=7B2024.02 | 53.82 | — | |
| VPG-C-LLaMA2#Param LLM=7B, Task Setting=Multi-image / Video Tasks2024.02 | 53.82 | — | |
| InstructBLIP-XLLLM Backbone=FlanT5, #Param LLM=3B2024.02 | 52.59 | — | |
| InstructBLIP-XL#Param LLM=3B, Task Setting=Multi-image / Video Tasks2024.02 | 52.59 | — | |
| Random Guessing2025.03 | 50.75 | — | |
| OtterLLM Backbone=MPT, #Param LLM=7B2024.02 | 49.54 | — | |
| Otter#Param LLM=7B, Task Setting=Multi-image / Video Tasks2024.02 | 49.54 | — | |
| Task ArithmeticBackbone=BEiT3-base2024.05 | 38.09 | — | |
| Ties-MergingBackbone=BEiT3-base2024.05 | 32.06 | — | |
| Weight AveragingBackbone=BEiT3-base2024.05 | 28 | — | |
| BERT-Attack (B&A)Pre-trained Model=ViLT, Attack Modality=Text Only2023.10 | — | 32.18 | |
| BERT-Attack (B&A)Pre-trained Model=BLIP, Attack Modality=Text Only2023.10 | — | 33.08 | |
| BSAPre-trained Model=ViLT, Attack Modality=Image Only2023.10 | — | 52.17 | |
| BSAPre-trained Model=BLIP, Attack Modality=Image Only2023.10 | — | 27.16 | |
| Co-AttackPre-trained Model=ViLT, Attack Modality=Multimodality2023.10 | — | 42.04 | |
| Co-AttackPre-trained Model=BLIP, Attack Modality=Multimodality2023.10 | — | 8.7 | |
| DRPre-trained Model=ViLT, Attack Modality=Image Only2023.10 | — | 21.58 | |
| DRPre-trained Model=BLIP, Attack Modality=Image Only2023.10 | — | 6.66 | |
| FDAPre-trained Model=ViLT, Attack Modality=Image Only2023.10 | — | 22.6 | |
| FDAPre-trained Model=BLIP, Attack Modality=Image Only2023.10 | — | 10.22 | |
| R&RPre-trained Model=ViLT, Attack Modality=Text Only2023.10 | — | 24.82 | |
| R&RPre-trained Model=BLIP, Attack Modality=Text Only2023.10 | — | 16.92 | |
| SSPPre-trained Model=ViLT, Attack Modality=Image Only2023.10 | — | 35.13 | |
| SSPPre-trained Model=BLIP, Attack Modality=Image Only2023.10 | — | 6.88 | |
| VLATTACKPre-trained Model=ViLT, Attack Modality=Multimodality2023.10 | — | 66.65 | |
| VLATTACKPre-trained Model=BLIP, Attack Modality=Multimodality2023.10 | — | 52.66 |