Colored MNIST
Benchmarks
TaskDatasetSOTA resultTrendResults
Colored-MNIST (CMNIST)
94.43Accuracy
42
Colored MNIST unbiased (test)
98.02Accuracy
28
Colored MNIST foreground color (test)
86.37Unbiased Accuracy
24
Colored MNIST (test)
75Accuracy
24
Colored MNIST a=4
20.44Mean Out-Sample W2
16
Colored MNIST a=3
18.39Mean Out-Sample W2
16
Colored MNIST a=2
18.23Mean Out-Sample W2
16
Colored MNIST a=1
17.62Mean Out-Sample W2
16
Colored MNIST a=0
20.29Mean Out-Sample W2
16
Colored MNIST (unbiased)
0.963Accuracy
16
Colored MNIST (bias-guiding)
100Accuracy
16
Colored MNIST DomainBed (leave-one-environment-out)
82Performance Score (+90%)
15
Colored MNIST (train)
88.9Accuracy
14
Colored MNIST 5.0% bias-conflict ratio
91.72BC Score
9
Colored MNIST 1.0% bias-conflict ratio
92.73BC Score
9
Colored MNIST
73.1Accuracy (+90% Threshold)
9
Colored-MNIST Level 4
100Accuracy (Red Biased)
8
Colored MNIST Non-IID partition
7.02FIDo
8
Colored MNIST IID partition
8.19FID (o)
8
Colored MNIST simulation (test)
0.519NLL
5
Colored MNIST background color, ratio 0.95
99.9Bias-Aligned Accuracy
5
Colored MNIST background color, ratio 0.98
99.97Bias-Aligned Accuracy
5
Colored MNIST background color, ratio 0.99
100Bias Aligned Score
5
Colored MNIST background color, ratio 0.995
1Aligned Accuracy
5
Colored MNIST (val)
1Validation Accuracy (Kendall's tau)
5