Language Modeling on GitHub (test)
2.42PerplexityKERPLE-log
Evaluation Results
| Method | Links | |
|---|---|---|
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 2.42 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 2.44 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 2.44 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 2.46 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 2.47 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 2.48 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 2.49 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 2.53 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 2.54 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 2.57 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 2.57 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 2.57 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 2.58 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 2.6 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 2.6 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 2.61 | |
| T5L_ex=1024, Training length=512, Embedding Type=learnable2022.12 | 2.61 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 2.62 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 2.64 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 2.65 | |
| KERPLEL_ex=2048, Training length=512, Embedding Type=learnable2022.12 | 2.65 | |
| T5L_ex=2048, Training length=512, Embedding Type=learnable2022.12 | 2.65 | |
| KERPLEL_ex=1024, Training length=512, Embedding Type=learnable2022.12 | 2.67 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 2.68 | |
| SandwichL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 2.69 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 2.7 | |
| SmoothedL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 2.7 | |
| KERPLEL_ex=4096, Training length=512, Embedding Type=learnable2022.12 | 2.7 | |
| KERPLE-logExtrapolation Length=81922022.05 | 2.71 | |
| SandwichL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 2.71 | |
| SandwichL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 2.73 | |
| KERPLE-logExtrapolation Length=40962022.05 | 2.74 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 2.74 | |
| SmoothedL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 2.74 | |
| KERPLE-logExtrapolation Length=163842022.05 | 2.75 | |
| KERPLEL_ex=8192, Training length=512, Embedding Type=learnable2022.12 | 2.75 | |
| KERPLE-powerExtrapolation Length=81922022.05 | 2.76 | |
| KERPLE-powerExtrapolation Length=163842022.05 | 2.76 | |
| T5L_ex=512, Training length=512, Embedding Type=learnable2022.12 | 2.76 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 2.77 | |
| T5Extrapolation Length=40962022.05 | 2.78 | |
| SmoothedL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 2.78 | |
| KERPLE-powerExtrapolation Length=40962022.05 | 2.79 | |
| SandwichL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 2.79 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 2.8 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 2.81 | |
| KERPLEL_ex=512, Training length=512, Embedding Type=learnable2022.12 | 2.81 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 2.82 | |
| RotaryL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 2.82 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 2.83 | |
| SmoothedL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 2.83 | |
| ALiBiL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 2.83 | |
| T5Extrapolation Length=20482022.05 | 2.84 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 2.84 | |
| KERPLE-logExtrapolation Length=20482022.05 | 2.86 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 2.88 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 2.88 | |
| SandwichL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 2.88 | |
| SmoothedL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 2.88 | |
| KERPLE-powerExtrapolation Length=20482022.05 | 2.9 | |
| T5L_ex=4096, Training length=512, Embedding Type=learnable2022.12 | 2.91 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 2.93 | |
| T5Extrapolation Length=81922022.05 | 2.95 | |
| ALiBiL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 2.97 | |
| ALiBiL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 3.01 | |
| ALiBiL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 3.01 | |
| ALiBiL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 3.01 | |
| ALiBiExtrapolation Length=163842022.05 | 3.02 | |
| T5Extrapolation Length=10242022.05 | 3.02 | |
| KERPLE-logExtrapolation Length=10242022.05 | 3.04 | |
| ALiBiExtrapolation Length=40962022.05 | 3.04 | |
| ALiBiExtrapolation Length=81922022.05 | 3.04 | |
| KERPLE-powerExtrapolation Length=10242022.05 | 3.07 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 3.1 | |
| ALiBiExtrapolation Length=20482022.05 | 3.13 | |
| ALiBiExtrapolation Length=10242022.05 | 3.15 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 3.16 | |
| T5Extrapolation Length=163842022.05 | 3.35 | |
| T5Extrapolation Length=5122022.05 | 3.38 | |
| KERPLE-logExtrapolation Length=5122022.05 | 3.4 | |
| KERPLE-powerExtrapolation Length=5122022.05 | 3.42 | |
| ALiBiExtrapolation Length=5122022.05 | 3.42 | |
| RotaryExtrapolation Length=5122022.05 | 3.44 | |
| T5L_ex=8192, Training length=512, Embedding Type=learnable2022.12 | 3.68 | |
| RotaryExtrapolation Length=10242022.05 | 3.86 | |
| RotaryL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 3.86 | |
| SinusoidalExtrapolation Length=5122022.05 | 4 | |
| SinusoidalL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 4 | |
| FOTContext Length=64K, Zero-shot=true2023.07 | 5.32 | |
| FOTContext Length=16K, Zero-shot=true2023.07 | 5.43 | |
| FOTContext Length=4K, Zero-shot=true2023.07 | 5.88 | |
| RotaryExtrapolation Length=20482022.05 | 5.94 | |
| RotaryL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 5.94 | |
| FOTContext Length=2K, Zero-shot=true2023.07 | 6.72 | |
| Transformer-XLContext Length=2K, Zero-shot=true2023.07 | 6.85 | |
| Memorizing TransformerContext Length=64K, Zero-shot=true2023.07 | 7.26 | |
| Memorizing TransformerContext Length=16K, Zero-shot=true2023.07 | 7.27 | |
| Memorizing TransformerContext Length=4K, Zero-shot=true2023.07 | 7.55 | |
| Memorizing TransformerContext Length=2K, Zero-shot=true2023.07 | 8.1 | |
| RotaryExtrapolation Length=40962022.05 | 11.1 |