Domain Generalization on DomainBed v1.0
82.3Average AccuracyCLIP S + CAD
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| CLIP S + CADModel Selection=oracle selection, Backbone=CLIP-ResNet502021.12 | 82.3 | — | — | — | — | — | — | |
| CLIP L + CADModel Selection=oracle selection, Backbone=CLIP-ViT-L/142021.12 | 81.6 | — | — | — | — | — | — | |
| CLIP S + BaseModel Selection=oracle selection, Backbone=CLIP-ResNet502021.12 | 81.3 | — | — | — | — | — | — | |
| CLIP SModel Selection=oracle selection, Backbone=CLIP-ResNet502021.12 | 81.1 | — | — | — | — | — | — | |
| CLIP SModel Selection=oracle selection, Backbone=CLIP-ResNet50, zero-shot=true2021.12 | 80.9 | — | — | — | — | — | — | |
| CLIP LModel Selection=oracle selection, Backbone=CLIP-ViT-L/142021.12 | 80.7 | — | — | — | — | — | — | |
| CDANNModel Selection=oracle selection, Backbone=ResNet-502021.12 | 79.9 | — | — | — | — | — | — | |
| DANNModel Selection=oracle selection, Backbone=ResNet-502021.12 | 79.7 | — | — | — | — | — | — | |
| MixupModel Selection=oracle selection, Backbone=ResNet-502021.12 | 78.1 | — | — | — | — | — | — | |
| VRExModel Selection=oracle selection, Backbone=ResNet-502021.12 | 78.1 | — | — | — | — | — | — | |
| CADModel Selection=oracle selection, Backbone=ResNet-502021.12 | 78 | — | — | — | — | — | — | |
| MMDModel Selection=oracle selection, Backbone=ResNet-502021.12 | 77.9 | — | — | — | — | — | — | |
| VL2V-SDBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 77.73 | 96.68 | 83.25 | 87.38 | 58.54 | 62.79 | 89.99 | |
| CORALModel Selection=oracle selection, Backbone=ResNet-502021.12 | 77.7 | — | — | — | — | — | — | |
| ERMModel Selection=oracle selection, Backbone=ResNet-502021.12 | 77.6 | — | — | — | — | — | — | |
| GroupDROModel Selection=oracle selection, Backbone=ResNet-502021.12 | 77.4 | — | — | — | — | — | — | |
| IRMModel Selection=oracle selection, Backbone=ResNet-502021.12 | 76.9 | — | — | — | — | — | — | |
| MIROBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 76.66 | 96.44 | 82.3 | 84.8 | 59.3 | 60.47 | 91 | |
| SAGMBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 75.49 | 94.31 | 82.05 | 83.4 | 58.64 | 59.05 | 89.74 | |
| MIROBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 73.72 | 95.6 | 81.32 | 83.36 | 54.3 | 54 | 89.32 | |
| DARTBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 73.06 | 93.43 | 80.38 | 80.93 | 51.24 | 59.32 | 89.25 | |
| EoQM (number of models)=5, S (relative network size)=1.1x, Backbone=ResNeXt-50 32x4d, Parameters=25M, Pre-trained=Instagram 1B images2024.10 | 72.9 | 93.5 | 80.3 | 80.3 | 55.6 | 54.8 | — | |
| VL2V-SDBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 72.73 | 95.67 | 82.67 | 85.44 | 41.18 | 58.71 | 89.5 | |
| EoAM (number of models)=6, S (relative network size)=6x, Backbone=ResNeXt-50 32x4d, Parameters=25M, Pre-trained=Instagram 1B images2024.10 | 72.7 | 93.2 | 80.4 | 80.2 | 55.2 | 54.6 | — | |
| SAGMBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 72.41 | 90.61 | 81.11 | 81.11 | 54.29 | 53.59 | 89.56 | |
| LP-FTBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 71.85 | 92.92 | 80.88 | 81.17 | 47.26 | 57.04 | 88.97 | |
| QT-DOGM (number of models)=1, S (relative network size)=1x, Backbone=ResNeXt-50 32x4d, Parameters=25M, Pre-trained=Instagram 1B images2024.10 | 71.8 | 92.9 | 79.2 | 78.9 | 54.1 | 53.9 | — | |
| SMAM (number of models)=1, S (relative network size)=1x, Backbone=ResNeXt-50 32x4d, Parameters=25M, Pre-trained=Instagram 1B images2024.10 | 71.6 | 92.7 | 79.7 | 78.6 | 53.3 | 53.5 | — | |
| CLIPoodBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 71.55 | 93.16 | 77.19 | 83.31 | 46.28 | 57.78 | 89.9 | |
| RISEBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 71.45 | 93.25 | 80.62 | 78.39 | 49.61 | 55.37 | 87.91 | |
| ERM Ens.M (number of models)=6, S (relative network size)=6x, Backbone=ResNeXt-50 32x4d, Parameters=25M, Pre-trained=Instagram 1B images2024.10 | 71.1 | 91.2 | 80.3 | 77.8 | 53.5 | 52.8 | — | |
| SWADBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 70.87 | 94.35 | 79.13 | 81.01 | 42.92 | 57.92 | 89.05 | |
| Zero-shotBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 70.6 | 96.5 | 82.3 | 82.4 | 34.1 | 57.7 | — | |
| DARTBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 69.76 | 91.45 | 77.04 | 77.35 | 46.41 | 56.53 | 88.65 | |
| FLYPBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 69.65 | 92.97 | 75.89 | 75.25 | 40.22 | 48.9 | 84.66 | |
| DINO + CADModel Selection=oracle selection, Backbone=DINO2021.12 | 69.6 | — | — | — | — | — | — | |
| DiWA†Weight selection=Uniform: M = 60, Init=LP [49], Backbone=ResNet502022.05 | 68 | 89 | 78.6 | 72.8 | 51.9 | 47.7 | — | |
| ERMM (number of models)=1, S (relative network size)=1x, Backbone=ResNeXt-50 32x4d, Parameters=25M, Pre-trained=Instagram 1B images2024.10 | 67.7 | 88.7 | 79 | 70.9 | 51.4 | 48.1 | — | |
| DiWAWeight selection=Uniform: M = 20, Init=LP [49], Backbone=ResNet502022.05 | 67.6 | 88.7 | 78.4 | 72.1 | 51.4 | 47.4 | — | |
| DiWAWeight selection=Restricted: M < 20, Init=LP [49], Backbone=ResNet502022.05 | 67.5 | 88 | 78.5 | 71.5 | 51.6 | 47.7 | — | |
| AdaClustModel Selection=Validation-Based Model Averaging (SWAD), Require domain labels=false, spectrum range=8 -> 520, clusters per class=52021.12 | 67.2 | 89.2 | 79.6 | 69.4 | 50.6 | 46.7 | — | |
| ENSWeight selection=Uniform: M = 20, Init=LP [49], Backbone=ResNet502022.05 | 67.2 | 88.1 | 78.5 | 71.7 | 50.8 | 47 | — | |
| LP-FTBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 67.14 | 86.28 | 77.1 | 69.72 | 36.04 | 49 | 84.72 | |
| ENSWeight selection=Uniform: M = 20, Init=Random, Backbone=ResNet50, Run_context=Our runs2022.05 | 67.1 | 88 | 78.7 | 70.5 | 51 | 47.4 | — | |
| DiWA†Weight selection=Uniform: M = 60, Init=Random, Backbone=ResNet502022.05 | 67.1 | 89 | 79.4 | 71.6 | 49 | 46.3 | — | |
| MAWeight selection=Uniform, Init=LP [49], Backbone=ResNet502022.05 | 67.1 | 87.8 | 78.5 | 71.5 | 51.4 | 46.6 | — | |
| DiWAWeight selection=Restricted: M < 20, Init=Random, Backbone=ResNet502022.05 | 67 | 87.9 | 79.2 | 70.5 | 50.5 | 46.7 | — | |
| SWADWeight selection=Overfit-aware, Init=Random, Backbone=ResNet502022.05 | 66.9 | 88.1 | 79.1 | 70.6 | 50 | 46.5 | — | |
| RISEBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 66.81 | 91.65 | 81.18 | 70.28 | 40.15 | 50.81 | 85.21 | |
| DENSWeight selection=Uniform: M = 6, Init=Random, Backbone=ResNet502022.05 | 66.8 | 87.6 | 78.5 | 70.8 | 49.2 | 47.7 | — | |
| DiWAWeight selection=Uniform: M = 20, Init=Random, Backbone=ResNet502022.05 | 66.8 | 88.8 | 79.1 | 71 | 48.9 | 46.1 | — | |
| MAWeight selection=Uniform, Init=Random, Backbone=ResNet50, Run_context=Our runs2022.05 | 66.6 | 87.9 | 78.4 | 70.3 | 49.9 | 46.4 | — | |
| ERMModel Selection=Validation-Based Model Averaging (SWAD), Require domain labels=false2021.12 | 66.5 | 87.8 | 78.8 | 69.8 | 50.2 | 46 | — | |
| MAWeight selection=Uniform, Init=Random, Backbone=ResNet502022.05 | 66.5 | 87.5 | 78.2 | 70.6 | 50.3 | 46 | — | |
| ERMWeight selection=N/A, Init=LP [49], Backbone=ResNet502022.05 | 65.6 | 85.9 | 78.1 | 69.4 | 50.4 | 44.3 | — | |
| CLIPoodBackbone=ViT-B/16, Initialization=CLIP, SWAD=false2023.10 | 65.23 | 79.61 | 78.32 | 67.51 | 35.68 | 47.72 | 82.52 | |
| AdaClustModel Selection=Leave-One-Domain-Out Validation, Require domain labels=false, spectrum range=512 -> 1024, clusters per class=52021.12 | 64.9 | 87 | 78.9 | 67.7 | 48.1 | 43.3 | — | |
| CoralWeight selection=N/A, Init=Random, Backbone=ResNet502022.05 | 64.6 | 86.2 | 78.8 | 68.7 | 47.6 | 41.5 | — | |
| ERMWeight selection=N/A, Init=Random, Backbone=ResNet50, Run_context=Our runs2022.05 | 64.6 | 85.5 | 77.6 | 67.4 | 48.3 | 44.1 | — | |
| DA-ERMModel Selection=Leave-One-Domain-Out Validation, Require domain labels=true2021.12 | 64.1 | 84.1 | 78 | 67.9 | 47.3 | 43.6 | — | |
| CORALModel Selection=Leave-One-Domain-Out Validation, Require domain labels=true2021.12 | 63.8 | 83.3 | 77.3 | 68.6 | 47.7 | 42.1 | — | |
| FLYPBackbone=ViT-B/16, Initialization=CLIP, SWAD=true2023.10 | 63.72 | 78.53 | 66.64 | 82.76 | 33.25 | 57.41 | 78.94 | |
| MixupModel Selection=Leave-One-Domain-Out Validation, Require domain labels=true2021.12 | 63.3 | 83.9 | 78.2 | 68.3 | 46.2 | 40.2 | — | |
| ERMWeight selection=N/A, Init=Random, Backbone=ResNet502022.05 | 63.3 | 85.5 | 77.5 | 66.5 | 46.1 | 40.9 | — | |
| MLDGModel Selection=Leave-One-Domain-Out Validation, Require domain labels=true2021.12 | 63 | 82.6 | 76.8 | 67.7 | 46 | 42.2 | — | |
| ERMModel Selection=Leave-One-Domain-Out Validation, Require domain labels=false2021.12 | 62.9 | 83.2 | 76.7 | 67.2 | 46.2 | 41.1 | — | |
| RSCModel Selection=Leave-One-Domain-Out Validation, Require domain labels=false2021.12 | 62.8 | 87.7 | 75.3 | 64.1 | 45.7 | 41.2 | — | |
| MMDModel Selection=Leave-One-Domain-Out Validation, Require domain labels=true2021.12 | 62.2 | 83.1 | 76.1 | 67.3 | 45.8 | 38.7 | — | |
| IRMModel Selection=Leave-One-Domain-Out Validation, Require domain labels=false2021.12 | 61.2 | 82.8 | 76.9 | 67 | 43.8 | 35.7 | — | |
| DROModel Selection=Leave-One-Domain-Out Validation, Require domain labels=false2021.12 | 60.4 | 83.3 | 77.3 | 66.8 | 42 | 33 | — | |
| C-DANNModel Selection=Leave-One-Domain-Out Validation, Require domain labels=true2021.12 | 60.1 | 81.4 | 73.8 | 64.2 | 40.9 | 29.5 | — |