Visual Commonsense Reasoning on VCR (test)
94AccuracyModCR
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ModCRTypes=fine-tune VLMs2023.05 | 94 | — | — | — | — | — | — | |
| ModCRTypes=frozen VLMs2023.05 | 93.6 | — | — | — | — | — | — | |
| ROBERTa-L2023.05 | 91.8 | — | — | — | — | — | — | |
| CALEC2023.05 | 90.5 | — | — | — | — | — | — | |
| OFA-L2023.05 | 89.4 | — | — | — | — | — | — | |
| Oscar-B2023.05 | 86 | — | — | — | — | — | — | |
| MVPTR2023.05 | 85.3 | — | — | — | — | — | — | |
| KVL-BERT_LARGEModel Scale=Large2020.12 | 78.6 | — | — | — | — | — | — | |
| VL-BERT_LARGEModel Scale=Large2020.12 | 78.4 | — | — | — | — | — | — | |
| KVL-BERT_LARGEModel Scale=Large2020.12 | 76.4 | — | — | — | — | — | — | |
| VL-BERT_LARGEModel Scale=Large2020.12 | 75.8 | — | — | — | — | — | — | |
| B2T22019.08 | 75.7 | — | — | — | — | — | — | |
| B2T22020.12 | 75.7 | — | — | — | — | — | — | |
| VL-BERT2019.08 | 74.8 | — | — | — | — | — | — | |
| ViLBERT2019.08 | 74.6 | — | — | — | — | — | — | |
| ViLBERT2020.12 | 74.6 | — | — | — | — | — | — | |
| Unicoder-VL2019.08 | 74.4 | — | — | — | — | — | — | |
| Unicoder-VL2020.12 | 74.4 | — | — | — | — | — | — | |
| Unicoder-VL2020.12 | 73.4 | — | — | — | — | — | — | |
| ViLBERT2020.12 | 73.3 | — | — | — | — | — | — | |
| VisualBERT2019.08 | 73.2 | — | — | — | — | — | — | |
| VisualBERT2020.12 | 73.2 | — | — | — | — | — | — | |
| B2T22020.12 | 72.6 | — | — | — | — | — | — | |
| VisualBERT2020.12 | 71.6 | — | — | — | — | — | — | |
| HGL2020.12 | 70.8 | — | — | — | — | — | — | |
| CCN2020.12 | 70.5 | — | — | — | — | — | — | |
| HGL2020.12 | 70.1 | — | — | — | — | — | — | |
| CCN2020.12 | 68.5 | — | — | — | — | — | — | |
| R2C2019.08 | 67.3 | — | — | — | — | — | — | |
| R2C2020.12 | 67.3 | — | — | — | — | — | — | |
| R2C2020.12 | 65.1 | — | — | — | — | — | — | |
| KVL-BERT_LARGEModel Scale=Large2020.12 | 60.3 | — | — | — | — | — | — | |
| VL-BERT_LARGEModel Scale=Large2020.12 | 59.7 | — | — | — | — | — | — | |
| VL-T5Pretraining Images=180K2021.02 | 58.9 | — | — | — | — | — | — | |
| UNITER BasePretraining Images=4M2021.02 | 58.2 | — | — | — | — | — | — | |
| VL-BERT2019.08 | 55.5 | — | — | — | — | — | — | |
| B2T22019.08 | 55 | — | — | — | — | — | — | |
| B2T22020.12 | 55 | — | — | — | — | — | — | |
| Unicoder-VL2019.08 | 54.9 | — | — | — | — | — | — | |
| Unicoder-VL2020.12 | 54.9 | — | — | — | — | — | — | |
| ViLBERT2019.08 | 54.8 | — | — | — | — | — | — | |
| ViLBERT2020.12 | 54.8 | — | — | — | — | — | — | |
| ViLBERTPretraining Images=3M2021.02 | 54.8 | — | — | — | — | — | — | |
| VisualBERT2019.08 | 52.4 | — | — | — | — | — | — | |
| VisualBERT2020.12 | 52.4 | — | — | — | — | — | — | |
| HGL2020.12 | 49.8 | — | — | — | — | — | — | |
| VL-BARTPretraining Images=180K2021.02 | 48.9 | — | — | — | — | — | — | |
| CCN2020.12 | 48.4 | — | — | — | — | — | — | |
| R2C2019.08 | 44 | — | — | — | — | — | — | |
| R2C2020.12 | 44 | — | — | — | — | — | — | |
| ChatGPT w/ VerbalizersComparison=vs BLIP-2 + LSKD2023.12 | 34.7 | — | — | — | 33.9 | 39.6 | 45 | |
| Mini-GPT4 (Vicuna-13B) + LSKDBackbone=Vicuna-13B, Framework=Mini-GPT4, Distillation=LSKD2023.12 | 34.3 | — | — | — | 53 | 34.2 | 49.1 | |
| BLIP-2 (FlanT5XXL-11B) + LSKDBackbone=FlanT5XXL-11B, Framework=BLIP-2, Distillation=LSKD2023.12 | 31.7 | — | — | — | 41 | 30.2 | 41.2 | |
| ChatGPT w/ VerbalizersComparison=vs Mini-GPT4 + LSKD2023.12 | 29.8 | — | — | — | 31.7 | 36.8 | 40.6 | |
| ALBEFSupervision=weakly supervised2022.05 | — | 72.9 | 74.5 | 54.7 | — | — | — | |
| B2T22019.08 | — | 72.6 | 75.7 | 55 | — | — | — | |
| B2T2Model=Single2019.08 | — | 72.6 | 75.7 | 55 | — | — | — | |
| B2T22022.05 | — | 74 | 77.1 | 57.1 | — | — | — | |
| ERNIE-VIL-LCode=Y2023.07 | — | 79.2 | 83.5 | 66.3 | — | — | — | |
| GPT4RoI-13BCode=Y, #Params=13B+2023.07 | — | 89.4 | 91 | 81.6 | — | — | — | |
| Human Performance2023.07 | — | 91 | 93 | 85 | — | — | — | |
| Hun Yuan-VCR@TencentCode=N2023.07 | — | 85.8 | 88 | 75.6 | — | — | — | |
| KS-MGSR@KDDI Research and SNAPCode=N2023.07 | — | 85.3 | 86.9 | 74.3 | — | — | — | |
| MERLOTCode=Y, #Params=223M2023.07 | — | 80.6 | 80.4 | 65.1 | — | — | — | |
| PEVL2022.05 | — | 76 | 76.7 | 58.6 | — | — | — | |
| R2C2019.08 | — | 65.1 | 67.3 | 44 | — | — | — | |
| R2CModel=Single2019.08 | — | 65.1 | 67.3 | 44 | — | — | — | |
| R2C2022.05 | — | 65.1 | 67.3 | 44 | — | — | — | |
| RESERVE-LCode=Y, #Params=644M2023.07 | — | 84 | 84.9 | 72 | — | — | — | |
| SP-VCR@ShopeeCode=N2023.07 | — | 83.6 | 88.6 | 74.4 | — | — | — | |
| TAB-VCR2022.05 | — | 70.4 | 71.7 | 50.5 | — | — | — | |
| Unicoder-VL2022.05 | — | 73.4 | 74.4 | 54.9 | — | — | — | |
| Unicoder-VLCode=Y2023.07 | — | 73.4 | 74.4 | 54.9 | — | — | — | |
| UNITER2022.05 | — | 75 | 77.2 | 58.2 | — | — | — | |
| UNITER-LCode=Y, #Params=303M2023.07 | — | 77.3 | 80.8 | 62.8 | — | — | — | |
| ViLBERTModel=Single2019.08 | — | 73.3 | 74.6 | 54.8 | — | — | — | |
| VILBERT2022.05 | — | 73.3 | 74.6 | 54.8 | — | — | — | |
| VILBERTCode=Y, #Params=221M2023.07 | — | 73.3 | 74.6 | 54.8 | — | — | — | |
| VILLA-LCode=Y2023.07 | — | 78.9 | 82.8 | 65.7 | — | — | — | |
| VisualBERTEarly Fusion=true, COCO Pre-training=true2019.08 | — | 71.6 | 73.2 | 52.4 | — | — | — | |
| VisualBERTModel=Single2019.08 | — | 71.6 | 73.2 | 52.4 | — | — | — | |
| VisualBERT2022.05 | — | 71.6 | 73.2 | 52.4 | — | — | — | |
| VL-BERTBackbone=BERT-Large, Pre-training=Full2019.08 | — | 75.8 | 78.4 | 59.7 | — | — | — | |
| VL-BERT2022.05 | — | 75.8 | 78.4 | 59.7 | — | — | — | |
| VLBERT-LCode=Y, #Params=383M2023.07 | — | 75.8 | 78.4 | 59.7 | — | — | — | |
| VLUA+@KuaishouCode=N2023.07 | — | 84.8 | 87 | 74 | — | — | — | |
| VQA-GNN-LCode=Y, #Params=1B+2023.07 | — | 85.2 | 86.6 | 74 | — | — | — |