Language Modeling on OpenWebText standard (test)
20.08PerplexityBase
Evaluation results
| Method | Links | |
|---|---|---|
| BaseBudget=24x, Configuration=(24 ⊗ 1), Params=24x, FLOPs=24x2026.02 | 20.08 | |
| BaseBudget=12x, Configuration=(12 ⊗ 1), Params=12x, FLOPs=12x2026.02 | 21.41 | |
| LoopFormerBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 21.98 | |
| TMLTBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 21.99 | |
| TMLT-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 23.17 | |
| LoopFormerBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 24.21 | |
| BaseBudget=6x, Configuration=(6 ⊗ 1), Params=6x, FLOPs=6x2026.02 | 24.28 | |
| Base-Loop-EE-ConsBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 24.41 | |
| Base-LoopBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 24.53 | |
| Naive-Loop-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 25.64 | |
| Naive-Loop-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 25.81 | |
| Base-Loop-EE-ConsBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 26.68 | |
| TMLT-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 27.11 | |
| Naive-Loop-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 28.38 | |
| LoopFormerBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 32.46 | |
| Base-Loop-EE-ConsBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 34.88 | |
| TMLT-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 37.84 |