Tasks
Benchmarks
TaskDatasetSOTA resultTrendResults
7 tasks zero-shot
72.79Mean Accuracy (Zero-shot)
223
26 few-shot tasks Random -> Random transfer setting (test)
48.95Accuracy
84
26 few-shot tasks Non-Class -> Class transfer setting (test)
0.528Accuracy
84
26 few-shot tasks Class -> Non-Class transfer setting (test)
45.54Accuracy
84
tasks 0-shot
68.58Accuracy
83
5 zero-shot tasks
79.48Accuracy
55
14 Tasks Merge
94.3Average Accuracy
51
Eight tasks zero-shot
60.31Accuracy (Zero-shot)
29
Zero-shot Tasks
73.99Task Avg Score
26
11 Tasks zero-shot
68.660-shot Average
26
Tasks Zero-shot (mean)
76.57mAcc
25
11 tasks average
83.83DSC
13
66 unseen tasks (test)
81.58Success Rate 1
10
350 tasks (train)
88.76Success Rate 1
10
9 Zero-shot Tasks (BoolQ, HellaSwag, LAMBADA, OBQA, PIQA, SIQA, WinoGrande, ARC-Easy, ARC-Challenge)
0.7110-shot Avg Accuracy
9
21 Tasks
56.421 Tasks Avg Score
9
27 tasks across 9 scene types
1,866Elo Rating
4
300 tasks (evaluation)
75.1Average Quality
4
In-Distribution Tasks Average
86.67Success Rate
3
5 Tasks Unseen Target Setup (train)
100Task 1 (Cola Can) Approach
3
5 Tasks Unseen Distractor Setup (train)
100Task 1 (cola can) App Success
3
5 Tasks Basic Setup (train)
100Task 1 (cola can) Approach Rate
3