Attributing Multimodal Foundation Model Errors on ImageNet misclassified samples (val)
85.76Avg Highest Confidence (0-50%)Greedy
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GreedyBackbone=CLIP ViT-L/142026.06 | 85.76 | — | — | 89.32 | 68.37 | 6.52 | 1,755.77 | 3.89 | |
| PhaseWinBackbone=CLIP ViT-L/142026.06 | 81.05 | — | — | 87.74 | 64.21 | 7.05 | 1,026.52 | 6.26 | |
| LIMABackbones=ImageBind [28], Sub-region division=Segment Anything2025.04 | 74.87 | 66.39 | 77.36 | 78.72 | 42.37 | — | — | — | |
| LIMABackbones=CLIP (ViT-L) [27], Sub-region division=Segment Anything2025.04 | 73.07 | 64.4 | 76.76 | 78.78 | 41.51 | — | — | — | |
| LIMABackbones=CLIP (ViT-L) [27], Sub-region division=SLICO2025.04 | 70.3 | 56.77 | 74.79 | 75.83 | 47.89 | — | — | — | |
| LIMABackbones=ImageBind [28], Sub-region division=SLICO2025.04 | 68.16 | 53.51 | 74.45 | 75.57 | 46.81 | — | — | — | |
| LIMABackbones=LanguageBind [29], Sub-region division=Segment Anything2025.04 | 63.99 | 54.01 | 67.53 | 69.55 | 34.75 | — | — | — | |
| Previous workBackbones=ImageBind [28]2025.04 | 58.65 | 44.36 | 61.57 | 63.48 | 39.29 | — | — | — | |
| RISEBackbone=CLIP ViT-L/142026.06 | 58.51 | — | — | 74.74 | 41.33 | 16.05 | 5,000 | 0.83 | |
| HSICBackbone=CLIP ViT-L/142026.06 | 57.57 | — | — | 71.24 | 40.35 | 15.18 | 1,536 | 2.63 | |
| Grad-ECLIPBackbone=CLIP ViT-L/142026.06 | 56.44 | — | — | 73.63 | 39.58 | 16.16 | — | — | |
| LIMABackbones=LanguageBind [29], Sub-region division=SLICO2025.04 | 53.11 | 38.43 | 58.24 | 59.78 | 34.83 | — | — | — | |
| RISEBackbones=ImageBind [28]2025.04 | 40.45 | 24.27 | 47.57 | 51.64 | 22.37 | — | — | — | |
| RISEBackbones=CLIP (ViT-L) [27]2025.04 | 39.85 | 27.1 | 45.58 | 48.79 | 18.27 | — | — | — | |
| IGOS++Backbone=CLIP ViT-L/142026.06 | 38.66 | — | — | 70.16 | 30.35 | 23.95 | — | — | |
| Grad-ECLIPBackbones=CLIP (ViT-L) [27]2025.04 | 34.04 | 22.02 | 37.67 | 41.23 | 13.97 | — | — | — | |
| GradientBackbone=CLIP ViT-L/142026.06 | 31.18 | — | — | 69.33 | 26.07 | 28.31 | — | — | |
| Previous workBackbones=CLIP (ViT-L) [27]2025.04 | 30.73 | 20.12 | 35.82 | 39.89 | 20.17 | — | — | — | |
| Previous workBackbones=LanguageBind [29]2025.04 | 29.33 | 20.5 | 33.6 | 35.06 | 18.12 | — | — | — | |
| Gradient IntegralBackbone=CLIP ViT-L/142026.06 | 28.17 | — | — | 68.99 | 24.68 | 29.84 | — | — | |
| EACBackbones=LanguageBind [29], Sub-region division=Segment Anything2025.04 | 28.02 | 14.04 | 37.55 | 42.45 | 17.21 | — | — | — | |
| HSIC-AttributionBackbones=ImageBind [28]2025.04 | 27.26 | 17.11 | 32.6 | 36.36 | 21.8 | — | — | — | |
| HSIC-AttributionBackbones=CLIP (ViT-L) [27]2025.04 | 27.23 | 21.48 | 30.51 | 34.62 | 10.62 | — | — | — | |
| ViT-CXBackbones=ImageBind [28]2025.04 | 25.93 | 12.99 | 35.32 | 42.61 | 11.29 | — | — | — | |
| ViT-CXBackbones=CLIP (ViT-L) [27]2025.04 | 23.67 | 12.68 | 31.37 | 39.02 | 12.36 | — | — | — | |
| RISEBackbones=LanguageBind [29]2025.04 | 19.02 | 9.67 | 25.3 | 29.2 | 15 | — | — | — | |
| EACBackbones=ImageBind [28], Sub-region division=Segment Anything2025.04 | 19.01 | 10.64 | 26.58 | 34.63 | 11.24 | — | — | — | |
| EACBackbones=CLIP (ViT-L) [27], Sub-region division=Segment Anything2025.04 | 18.61 | 10.04 | 25.61 | 33.97 | 10.86 | — | — | — | |
| HSIC-AttributionBackbones=LanguageBind [29]2025.04 | 13.8 | 7.87 | 17.79 | 21.12 | 8.61 | — | — | — | |
| ViT-CXBackbones=LanguageBind [29]2025.04 | 11.23 | 4.14 | 16.28 | 23.48 | 8.16 | — | — | — |