List your company
ResearchFormerly WizWand

Multimodal Classification on UPMC Food-101 (test)

94.1AccuracyMMBT*
65.177672.686380.19587.7037Mar 6, 2023Jun 11, 2023Sep 16, 2023Dec 22, 2023Mar 28, 2024Jul 3, 2024Oct 9, 2024
Updated 8mo ago

Evaluation results

MethodLinks
MMBT*Updated Param. (Million)=196.5, Train Memory Usage (GB)=37.87, Inference Memory Usage (GB)=3.48, Backbone=bert-base / vit-base2023.0494.1
MBT*Updated Param. (Million)=196.0, Train Memory Usage (GB)=38, Inference Memory Usage (GB)=4.06, Backbone=bert-base / vit-base2023.0493.56
LateConcatUpdated Param. (Million)=196.0, Train Memory Usage (GB)=38.54, Inference Memory Usage (GB)=3.36, Backbone=bert-base / vit-base2023.0493.29
PMF-large (M=4, Lf=22)Updated Param. (Million)=4.5, Train Memory Usage (GB)=18.44, Inference Memory Usage (GB)=6.42, Backbone=bert-large / vit-large, M (prompt length)=4, Lf (starting fusion layer)=222023.0491.68
PMF (M=4, Lf=10)Updated Param. (Million)=2.5, Train Memory Usage (GB)=12.84, Inference Memory Usage (GB)=4.08, Backbone=bert-base / vit-base, M (prompt length)=4, Lf (starting fusion layer)=102023.0491.51
DCPMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.1089.12
P-LateConcatUpdated Param. (Million)=0.3, Train Memory Usage (GB)=30.82, Inference Memory Usage (GB)=3.43, Backbone=bert-base / vit-base, Prompt Length=102023.0489.03
BERTUpdated Param. (Million)=109.0, Train Memory Usage (GB)=30.82, Inference Memory Usage (GB)=2.79, Backbone=bert-base2023.0487.44
MaPLeMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.1087.35
MMPMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.1087.16
CoOpMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.1087.02
P-MMBTUpdated Param. (Million)=0.9, Train Memory Usage (GB)=30.9, Inference Memory Usage (GB)=3.48, Backbone=bert-base / vit-base, Prompt Length=102023.0486.58
Input-level promptsMissing rate η=70%, Training Image=30%, Training Text=100%, Testing Image=30%, Testing Text=100%2023.0386.18
Attention-level promptsMissing rate η=70%, Training Image=30%, Training Text=100%, Testing Image=30%, Testing Text=100%2023.0386.05
BlindPromptUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=29.57, Inference Memory Usage (GB)=3.65, Backbone=bert-base / vit-base, Prompt Length=202023.0484.56
PromptFuseUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=29.57, Inference Memory Usage (GB)=3.55, Backbone=bert-base / vit-base, Prompt Length=202023.0482.21
DePTMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.1082.14
P-BERTUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=28.13, Inference Memory Usage (GB)=2.99, Backbone=bert-base, Prompt Length=102023.0481.07
Input-level promptsMissing rate η=70%, Training Image=65%, Training Text=65%, Testing Image=65%, Testing Text=65%2023.0379.08
LinearUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=3.76, Inference Memory Usage (GB)=3.23, Backbone=bert-base / vit-base2023.0478.96
Attention-level promptsMissing rate η=70%, Training Image=65%, Training Text=65%, Testing Image=65%, Testing Text=65%2023.0378.09
BaselineMissing rate η=70%, Training Image=30%, Training Text=100%, Testing Image=30%, Testing Text=100%2023.0376.66
ViTUpdated Param. (Million)=86.5, Train Memory Usage (GB)=9.36, Inference Memory Usage (GB)=1.99, Backbone=vit-base2023.0474.69
Input-level promptsMissing rate η=70%, Training Image=100%, Training Text=30%, Testing Image=100%, Testing Text=30%2023.0374.53
Attention-level promptsMissing rate η=70%, Training Image=100%, Training Text=30%, Testing Image=100%, Testing Text=30%2023.0372.57
VPTUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=6.12, Inference Memory Usage (GB)=2.01, Backbone=vit-base, Prompt Length=102023.0472.55
BaselineMissing rate η=70%, Training Image=65%, Training Text=65%, Testing Image=65%, Testing Text=65%2023.0369.25
BaselineMissing rate η=70%, Training Image=100%, Training Text=30%, Testing Image=100%, Testing Text=30%2023.0366.29