Language Modeling on OpenWebText2 (test)
16.2PerplexityKERPLE-log
Evaluation results
| Method | Links | |
|---|---|---|
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 16.2 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 16.4 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 16.4 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 16.5 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 16.5 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 16.5 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 16.6 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 16.7 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 16.7 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 16.9 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 16.9 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 17.5 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 17.5 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 17.8 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 17.8 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 18 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 18.6 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 18.7 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 18.8 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 19 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 19.1 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 19.2 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 19.2 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 19.2 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 19.2 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 19.3 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 19.3 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 19.3 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 19.5 | |
| KERPLE-logExtrapolation Length=40962022.05 | 21.2 | |
| KERPLE-logExtrapolation Length=81922022.05 | 21.3 | |
| KERPLE-logExtrapolation Length=163842022.05 | 21.4 | |
| KERPLE-powerExtrapolation Length=40962022.05 | 21.5 | |
| KERPLE-logExtrapolation Length=20482022.05 | 21.6 | |
| KERPLE-powerExtrapolation Length=81922022.05 | 21.6 | |
| KERPLE-powerExtrapolation Length=163842022.05 | 21.6 | |
| T5Extrapolation Length=20482022.05 | 21.7 | |
| KERPLE-powerExtrapolation Length=20482022.05 | 21.9 | |
| T5Extrapolation Length=10242022.05 | 21.9 | |
| KERPLEL_ex=2048, Training length=512, Embedding Type=learnable2022.12 | 21.9 | |
| KERPLE-logExtrapolation Length=10242022.05 | 22 | |
| KERPLEL_ex=1024, Training length=512, Embedding Type=learnable2022.12 | 22 | |
| KERPLE-powerExtrapolation Length=10242022.05 | 22.1 | |
| KERPLEL_ex=4096, Training length=512, Embedding Type=learnable2022.12 | 22.1 | |
| ALiBiExtrapolation Length=40962022.05 | 22.2 | |
| T5L_ex=1024, Training length=512, Embedding Type=learnable2022.12 | 22.2 | |
| ALiBiExtrapolation Length=81922022.05 | 22.3 | |
| KERPLEL_ex=8192, Training length=512, Embedding Type=learnable2022.12 | 22.3 | |
| ALiBiExtrapolation Length=10242022.05 | 22.4 | |
| ALiBiExtrapolation Length=20482022.05 | 22.5 | |
| ALiBiExtrapolation Length=163842022.05 | 22.5 | |
| T5Extrapolation Length=40962022.05 | 22.5 | |
| KERPLEL_ex=512, Training length=512, Embedding Type=learnable2022.12 | 22.6 | |
| T5L_ex=512, Training length=512, Embedding Type=learnable2022.12 | 22.6 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 22.7 | |
| ALiBiL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 22.8 | |
| SandwichL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 23 | |
| RotaryL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 23 | |
| T5L_ex=2048, Training length=512, Embedding Type=learnable2022.12 | 23 | |
| SmoothedL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 23.1 | |
| SmoothedL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 23.2 | |
| SmoothedL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 23.2 | |
| SandwichL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 23.3 | |
| ALiBiL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 23.3 | |
| SandwichL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 23.5 | |
| ALiBiL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 23.5 | |
| ALiBiL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 23.5 | |
| ALiBiL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 23.5 | |
| SmoothedL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 23.6 | |
| T5Extrapolation Length=5122022.05 | 23.7 | |
| SandwichL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 23.8 | |
| KERPLE-logExtrapolation Length=5122022.05 | 23.9 | |
| KERPLE-powerExtrapolation Length=5122022.05 | 23.9 | |
| ALiBiExtrapolation Length=5122022.05 | 23.9 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 24 | |
| SmoothedL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 24 | |
| RotaryExtrapolation Length=5122022.05 | 24.2 | |
| SandwichL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 24.7 | |
| T5Extrapolation Length=81922022.05 | 25.5 | |
| SinusoidalL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 26 | |
| T5L_ex=4096, Training length=512, Embedding Type=learnable2022.12 | 26.8 | |
| T5Extrapolation Length=163842022.05 | 31.4 | |
| RotaryExtrapolation Length=10242022.05 | 32.8 | |
| SinusoidalExtrapolation Length=5122022.05 | 33 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 37.1 | |
| T5L_ex=8192, Training length=512, Embedding Type=learnable2022.12 | 38.6 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 50.8 | |
| RotaryL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 61 | |
| RotaryExtrapolation Length=20482022.05 | 62.4 | |
| RotaryL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 96 | |
| RotaryExtrapolation Length=40962022.05 | 111 | |
| RotaryExtrapolation Length=81922022.05 | 185 | |
| RotaryL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 232 | |
| RotaryExtrapolation Length=163842022.05 | 269 | |
| RotaryL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 343 | |
| SinusoidalExtrapolation Length=10242022.05 | 750 | |
| SinusoidalExtrapolation Length=20482022.05 | 5,507 | |
| SinusoidalExtrapolation Length=40962022.05 | 14,039 | |
| SinusoidalL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 14,168 | |
| SinusoidalL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 20,370 |