List your company
ResearchFormerly WizWand

Image Classification on ImageNet Distribution Shifts Summary

86.54Avg Shifts ScoreBest model on each test set (oracle)
42.485653.922865.3676.7972Sep 4, 2021Nov 5, 2021Jan 7, 2022Mar 11, 2022May 12, 2022Jul 14, 2022Sep 15, 2022
Updated 8mo ago

Evaluation results

MethodLinks
Best model on each test set (oracle)Backbone=BASIC-L, Evaluation Protocol=fine-tuned, Model Selection Strategy=oracle (best on test set)2022.0386.54———
Greedy soupBackbone=BASIC-L, Evaluation Protocol=fine-tuned, Model Selection Strategy=greedy soup2022.0386.4———
Greedy ensembleBackbone=BASIC-L, Evaluation Protocol=fine-tuned, Model Selection Strategy=greedy ensemble2022.0386.2———
Best model on held out val setBackbone=BASIC-L, Evaluation Protocol=fine-tuned, Model Selection Strategy=best on val set2022.0385.63———
CoCaBackbone=CoCa, Evaluation Protocol=zero-shot2022.0385.54———
ViT-G/14 greedy soupBackbone=ViT/G-14, Model Selection Strategy=greedy soup2022.0385.02———
BASIC-LBackbone=BASIC-L, Evaluation Protocol=zero-shot2022.0384.06———
WiSE-FTModel=CLIP ViT-L/14@336px, Evaluation Protocol=End-to-End, Mixing Coefficient (alpha)=optimal2021.0977.481.9——
WiSE-FTModel=CLIP ViT-L/14@336px, Evaluation Protocol=End-to-End, Mixing Coefficient (alpha)=0.52021.0976.981.8——
WiSE-FTModel=CLIP ViT-L/14@336px, Evaluation Protocol=Linear Classifier, Mixing Coefficient (alpha)=0.52021.0975.979.8——
WiSE-FTModel=CLIP ViT-L/14@336px, Evaluation Protocol=Linear Classifier, Mixing Coefficient (alpha)=optimal2021.0975.980.2——
Zero-shotModel=CLIP ViT-L/14@336px, Evaluation Protocol=Zero-shot, Source=PyTorch2021.0973.475——
Zero-shotModel=CLIP ViT-L/14@336px, Evaluation Protocol=Zero-shot, Source=[82]2021.0973.374.8——
Fine-tuned LCModel=CLIP ViT-L/14@336px, Evaluation Protocol=Linear Classifier, Implementation=ours2021.0972.678.9——
Fine-tuned LCModel=CLIP ViT-L/14@336px, Evaluation Protocol=Linear Classifier, Source=[82]2021.0971.878.6——
Fine-tuned E2EModel=CLIP ViT-L/14@336px, Evaluation Protocol=End-to-End, Implementation=ours2021.0968.677.4——
TPT + CoOpBackbone=ViT-B/162022.0964.99—62.83—
TPT + CoCoOpBackbone=ViT-B/162022.0964.3—62.61—
TPTBackbone=ViT-B/162022.0962.44—60.81—
CoCoOpBackbone=ViT-B/162022.0962.13—59.91—
CoOpBackbone=ViT-B/162022.0961.72—59.28—
Prompt EnsembleBackbone=ViT-B/162022.0961.2—59.42—
CLIPBackbone=ViT-B/162022.0959.11—57.2—
TPT + CoOpBackbone=ResNet-502022.0949.55—45.75—
TPT + CoCoOpBackbone=ResNet-502022.0948.45—44.83—
TPTBackbone=ResNet-502022.0947.26—43.89—
CoCoOpBackbone=ResNet-502022.0946.81—42.82—
CoOpBackbone=ResNet-502022.0946.61—42.43—
Prompt EnsembleBackbone=ResNet-502022.0946.43—43.09—
CLIPBackbone=ResNet-502022.0944.18—40.69—
DAREBackbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———58.45
DAREBackbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———53.38
E2E-FTBackbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———53.7
Linear ProbingBackbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———57.39
MERGETUNEBackbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———59.66
MERGETUNEBackbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———62.29
MERGETUNE + Weight ens.Backbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———60.23
MERGETUNE + Weight ens.Backbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———62.9
TIESBackbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———58.76
TIESBackbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———60.33
VRFBackbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———58.87
VRFBackbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———61.72
Weight ens.Backbone=CLIP ViT-B/16, Protocol=Linear Probing2026.01———58.56
Weight ens.Backbone=CLIP ViT-B/16, Protocol=E2E-FT2026.01———60.64
Zero-shot (CLIP)Backbone=CLIP ViT-B/16, Protocol=Zero-shot2026.01———58.42