Multimodal Classification on SNLI-VE (test)
81.64AccuracyALBEF+VAWI+SBS
Evaluation Results
| Method | Links | |
|---|---|---|
| ALBEF+VAWI+SBStraining_data_percentage=100%2022.12 | 81.64 | |
| ALBEFtraining_data_percentage=100%2022.12 | 80.91 | |
| ALBEF+VAWI+SBStraining_data_percentage=50%2022.12 | 76.14 | |
| ALBEFtraining_data_percentage=50%2022.12 | 75.47 | |
| MMBT*Updated Param. (Million)=196.5, Train Memory Usage (GB)=37.87, Inference Memory Usage (GB)=3.48, Backbone=bert-base / vit-base2023.04 | 74.69 | |
| MBT*Updated Param. (Million)=196.0, Train Memory Usage (GB)=38, Inference Memory Usage (GB)=4.06, Backbone=bert-base / vit-base2023.04 | 74.02 | |
| PMF-large (M=4, Lf=22)Updated Param. (Million)=4.5, Train Memory Usage (GB)=18.44, Inference Memory Usage (GB)=6.42, Backbone=bert-large / vit-large, M (prompt length)=4, Lf (starting fusion layer)=222023.04 | 72.1 | |
| PMF (M=4, Lf=10)Updated Param. (Million)=2.5, Train Memory Usage (GB)=12.84, Inference Memory Usage (GB)=4.08, Backbone=bert-base / vit-base, M (prompt length)=4, Lf (starting fusion layer)=102023.04 | 71.92 | |
| LateConcatUpdated Param. (Million)=196.0, Train Memory Usage (GB)=38.54, Inference Memory Usage (GB)=3.36, Backbone=bert-base / vit-base2023.04 | 70.01 | |
| BERTUpdated Param. (Million)=109.0, Train Memory Usage (GB)=30.82, Inference Memory Usage (GB)=2.79, Backbone=bert-base2023.04 | 69.82 | |
| ALBEF+VAWI+SBStraining_data_percentage=20%2022.12 | 68.23 | |
| P-MMBTUpdated Param. (Million)=0.9, Train Memory Usage (GB)=30.9, Inference Memory Usage (GB)=3.48, Backbone=bert-base / vit-base, Prompt Length=102023.04 | 67.58 | |
| ALBEFtraining_data_percentage=20%2022.12 | 67.52 | |
| ALBEF+VAWI+SBStraining_data_percentage=10%2022.12 | 65.94 | |
| BlindPromptUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=29.57, Inference Memory Usage (GB)=3.65, Backbone=bert-base / vit-base, Prompt Length=202023.04 | 65.54 | |
| ALBEFtraining_data_percentage=10%2022.12 | 65.46 | |
| PromptFuseUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=29.57, Inference Memory Usage (GB)=3.55, Backbone=bert-base / vit-base, Prompt Length=202023.04 | 64.53 | |
| P-BERTUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=28.13, Inference Memory Usage (GB)=2.99, Backbone=bert-base, Prompt Length=102023.04 | 63.28 | |
| P-LateConcatUpdated Param. (Million)=0.3, Train Memory Usage (GB)=30.82, Inference Memory Usage (GB)=3.43, Backbone=bert-base / vit-base, Prompt Length=102023.04 | 63.05 | |
| LinearUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=3.76, Inference Memory Usage (GB)=3.23, Backbone=bert-base / vit-base2023.04 | 50.05 | |
| ViTUpdated Param. (Million)=86.5, Train Memory Usage (GB)=9.36, Inference Memory Usage (GB)=1.99, Backbone=vit-base2023.04 | 33.33 | |
| VPTUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=6.12, Inference Memory Usage (GB)=2.01, Backbone=vit-base, Prompt Length=102023.04 | 33.33 |