Zero-shot Reasoning on Language Task Suite (COPA, HS, LB, OBQA, PIQA, Race, SciQ, ARC, SIQA, WG)
68COPALoopFormer
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LoopFormerBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 68 | 31 | 32.35 | 25.4 | 63.06 | 28.23 | 66.3 | 31.78 | 37.77 | 53.43 | 43.73 | |
| BaseBudget=12x, Configuration=(12 ⊗ 1), Params=12x, FLOPs=12x2026.02 | 67 | 32.72 | 37.78 | 26.2 | 64.69 | 29.86 | 69.1 | 32.29 | 38.38 | 51.3 | 44.93 | |
| Naive-Loop-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 66 | 29.68 | 31.52 | 27 | 62.24 | 28.33 | 65.5 | 31.64 | 36.64 | 50.59 | 42.91 | |
| Base-Loop-EE-ConsBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 66 | 30.54 | 32.19 | 26.8 | 62.13 | 28.23 | 64.9 | 31.45 | 37.4 | 52.88 | 43.25 | |
| LoopFormerBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 66 | 32.3 | 38.27 | 26.8 | 63.33 | 30.81 | 68 | 32.71 | 37.97 | 51.94 | 44.81 | |
| TMLTBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 65 | 32.34 | 39.06 | 27.8 | 63.11 | 29.67 | 69.8 | 31.67 | 36.95 | 51.54 | 44.69 | |
| TMLT-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 65 | 31.03 | 35.14 | 28.4 | 63.11 | 28.8 | 68.8 | 31.74 | 37.41 | 50.83 | 44.03 | |
| Naive-Loop-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 65 | 29.35 | 31.28 | 26 | 61.75 | 28.32 | 64.4 | 31.15 | 36.8 | 51.85 | 42.59 | |
| BaseBudget=6x, Configuration=(6 ⊗ 1), Params=6x, FLOPs=6x2026.02 | 64 | 30.45 | 33.45 | 25.6 | 62.51 | 28.04 | 67.5 | 31.07 | 36.18 | 48.54 | 42.73 | |
| Base-Loop-EE-ConsBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 63 | 29.72 | 25.6 | 27 | 61.26 | 28.04 | 58.4 | 30.06 | 36.23 | 51.7 | 41.1 | |
| Naive-Loop-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 63 | 28.64 | 27.28 | 24.6 | 61.48 | 26.41 | 62.3 | 29.91 | 35.41 | 50.67 | 40.97 | |
| LoopFormerBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 63 | 28.69 | 26.14 | 26.6 | 60.1 | 25.74 | 58.6 | 29.29 | 35.26 | 50.2 | 40.36 | |
| BaseBudget=24x, Configuration=(24 ⊗ 1), Params=24x, FLOPs=24x2026.02 | 61 | 35.04 | 41.96 | 27.6 | 66 | 29 | 70.1 | 33.43 | 38.18 | 50.4 | 45.27 | |
| Base-LoopBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 61 | 30.46 | 34.68 | 27 | 63.22 | 28.8 | 63.7 | 31.71 | 38.43 | 49.8 | 42.88 | |
| TMLT-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 60 | 29.88 | 27.73 | 26.4 | 62.02 | 28.8 | 61.4 | 30.59 | 36.69 | 51.54 | 41.5 | |
| Base-Loop-EE-ConsBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 59 | 28.28 | 18.49 | 25.6 | 59.52 | 25.16 | 53.8 | 28.94 | 34.65 | 52.72 | 38.62 | |
| TMLT-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 57 | 28.18 | 17.09 | 25.08 | 58.68 | 26.41 | 50.01 | 28.26 | 34.9 | 50.27 | 37.59 |