Multimodal Classification on UPMC Food-101 (test)
94.1AccuracyMMBT*
Evaluation Results
| Method | Links | |
|---|---|---|
| MMBT*Updated Param. (Million)=196.5, Train Memory Usage (GB)=37.87, Inference Memory Usage (GB)=3.48, Backbone=bert-base / vit-base2023.04 | 94.1 | |
| MBT*Updated Param. (Million)=196.0, Train Memory Usage (GB)=38, Inference Memory Usage (GB)=4.06, Backbone=bert-base / vit-base2023.04 | 93.56 | |
| LateConcatUpdated Param. (Million)=196.0, Train Memory Usage (GB)=38.54, Inference Memory Usage (GB)=3.36, Backbone=bert-base / vit-base2023.04 | 93.29 | |
| PMF-large (M=4, Lf=22)Updated Param. (Million)=4.5, Train Memory Usage (GB)=18.44, Inference Memory Usage (GB)=6.42, Backbone=bert-large / vit-large, M (prompt length)=4, Lf (starting fusion layer)=222023.04 | 91.68 | |
| PMF (M=4, Lf=10)Updated Param. (Million)=2.5, Train Memory Usage (GB)=12.84, Inference Memory Usage (GB)=4.08, Backbone=bert-base / vit-base, M (prompt length)=4, Lf (starting fusion layer)=102023.04 | 91.51 | |
| DCPMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.10 | 89.12 | |
| P-LateConcatUpdated Param. (Million)=0.3, Train Memory Usage (GB)=30.82, Inference Memory Usage (GB)=3.43, Backbone=bert-base / vit-base, Prompt Length=102023.04 | 89.03 | |
| BERTUpdated Param. (Million)=109.0, Train Memory Usage (GB)=30.82, Inference Memory Usage (GB)=2.79, Backbone=bert-base2023.04 | 87.44 | |
| MaPLeMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.10 | 87.35 | |
| MMPMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.10 | 87.16 | |
| CoOpMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.10 | 87.02 | |
| P-MMBTUpdated Param. (Million)=0.9, Train Memory Usage (GB)=30.9, Inference Memory Usage (GB)=3.48, Backbone=bert-base / vit-base, Prompt Length=102023.04 | 86.58 | |
| Input-level promptsMissing rate η=70%, Training Image=30%, Training Text=100%, Testing Image=30%, Testing Text=100%2023.03 | 86.18 | |
| Attention-level promptsMissing rate η=70%, Training Image=30%, Training Text=100%, Testing Image=30%, Testing Text=100%2023.03 | 86.05 | |
| BlindPromptUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=29.57, Inference Memory Usage (GB)=3.65, Backbone=bert-base / vit-base, Prompt Length=202023.04 | 84.56 | |
| PromptFuseUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=29.57, Inference Memory Usage (GB)=3.55, Backbone=bert-base / vit-base, Prompt Length=202023.04 | 82.21 | |
| DePTMissing rate η=50%, Image presence (%)=50%, Text presence (%)=100%2024.10 | 82.14 | |
| P-BERTUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=28.13, Inference Memory Usage (GB)=2.99, Backbone=bert-base, Prompt Length=102023.04 | 81.07 | |
| Input-level promptsMissing rate η=70%, Training Image=65%, Training Text=65%, Testing Image=65%, Testing Text=65%2023.03 | 79.08 | |
| LinearUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=3.76, Inference Memory Usage (GB)=3.23, Backbone=bert-base / vit-base2023.04 | 78.96 | |
| Attention-level promptsMissing rate η=70%, Training Image=65%, Training Text=65%, Testing Image=65%, Testing Text=65%2023.03 | 78.09 | |
| BaselineMissing rate η=70%, Training Image=30%, Training Text=100%, Testing Image=30%, Testing Text=100%2023.03 | 76.66 | |
| ViTUpdated Param. (Million)=86.5, Train Memory Usage (GB)=9.36, Inference Memory Usage (GB)=1.99, Backbone=vit-base2023.04 | 74.69 | |
| Input-level promptsMissing rate η=70%, Training Image=100%, Training Text=30%, Testing Image=100%, Testing Text=30%2023.03 | 74.53 | |
| Attention-level promptsMissing rate η=70%, Training Image=100%, Training Text=30%, Testing Image=100%, Testing Text=30%2023.03 | 72.57 | |
| VPTUpdated Param. (Million)=< 0.1, Train Memory Usage (GB)=6.12, Inference Memory Usage (GB)=2.01, Backbone=vit-base, Prompt Length=102023.04 | 72.55 | |
| BaselineMissing rate η=70%, Training Image=65%, Training Text=65%, Testing Image=65%, Testing Text=65%2023.03 | 69.25 | |
| BaselineMissing rate η=70%, Training Image=100%, Training Text=30%, Testing Image=100%, Testing Text=30%2023.03 | 66.29 |