Language Modeling on ARXIV (test)
2.36PPLBRECT:FIXED:SKIP
Evaluation results
| Method | Links | |
|---|---|---|
| BRECT:FIXED:SKIPeval seq. length=4096, window length=512, number params=196M, TPUv4 hours (k)=0.82023.06 | 2.36 | |
| BST:MF:UNSTRUCT-Leval seq. length=4096, window length=512, number params=388M, TPUv4 hours (k)=0.82023.06 | 2.41 | |
| BST:MF:UNSTRUCTeval seq. length=4096, window length=512, number params=221M, TPUv4 hours (k)=0.52023.06 | 2.44 | |
| BST:MF:S4-Leval seq. length=4096, window length=512, number params=383M, TPUv4 hours (k)=0.82023.06 | 2.46 | |
| BST:SH:UNSTRUCT-Leval seq. length=4096, window length=512, number params=371M, TPUv4 hours (k)=0.82023.06 | 2.46 | |
| TRSF-XL:2048eval seq. length=2048, window length=2048, number params=190M, TPUv4 hours (k)=0.82023.06 | 2.48 | |
| BST:MF:S4eval seq. length=4096, window length=512, number params=217M, TPUv4 hours (k)=0.52023.06 | 2.48 | |
| BST:SH:UNSTRUCTeval seq. length=4096, window length=512, number params=206M, TPUv4 hours (k)=0.52023.06 | 2.49 | |
| BST:SH:S4-Leval seq. length=4096, window length=512, number params=366M, TPUv4 hours (k)=0.82023.06 | 2.49 | |
| BST:SH:S4eval seq. length=4096, window length=512, number params=202M, TPUv4 hours (k)=0.52023.06 | 2.51 | |
| GSS-HYBRID-Leval seq. length=4096, window length=512, number params=373M, TPUv4 hours (k)=0.82023.06 | 2.51 | |
| BST:MH:S4eval seq. length=4096, window length=512, number params=218M, TPUv4 hours (k)=0.82023.06 | 2.52 | |
| SLIDE: 12Leval seq. length=4096, window length=512, number params=190M, TPUv4 hours (k)=0.52023.06 | 2.69 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 4.45 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 4.47 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 4.5 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 4.53 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 4.54 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 4.55 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 4.62 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 4.65 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 4.65 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 4.74 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 4.75 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 4.76 | |
| KERPLEL_ex=2048, Training length=512, Embedding Type=learnable2022.12 | 4.83 | |
| KERPLEL_ex=4096, Training length=512, Embedding Type=learnable2022.12 | 4.84 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 4.87 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.05 | 4.87 | |
| KERPLEL_ex=8192, Training length=512, Embedding Type=learnable2022.12 | 4.9 | |
| T5L_ex=1024, Training length=512, Embedding Type=learnable2022.12 | 4.91 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 4.92 | |
| T5L_ex=2048, Training length=512, Embedding Type=learnable2022.12 | 4.92 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.05 | 4.94 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.05 | 4.94 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 4.94 | |
| KERPLEL_ex=1024, Training length=512, Embedding Type=learnable2022.12 | 4.95 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 4.97 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.05 | 4.97 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.05 | 4.98 | |
| KERPLE-logExtrapolation Length=81922022.05 | 5.01 | |
| SandwichL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 5.02 | |
| SandwichL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 5.05 | |
| KERPLE-powerExtrapolation Length=81922022.05 | 5.06 | |
| KERPLE-logExtrapolation Length=163842022.05 | 5.07 | |
| KERPLE-powerExtrapolation Length=163842022.05 | 5.07 | |
| SmoothedL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 5.13 | |
| SandwichL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 5.15 | |
| SmoothedL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 5.15 | |
| T5L_ex=512, Training length=512, Embedding Type=learnable2022.12 | 5.16 | |
| KERPLE-logExtrapolation Length=40962022.05 | 5.2 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 5.2 | |
| T5Extrapolation Length=20482022.05 | 5.21 | |
| KERPLE-logExtrapolation Length=20482022.05 | 5.22 | |
| KERPLEL_ex=512, Training length=512, Embedding Type=learnable2022.12 | 5.22 | |
| KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 5.23 | |
| KERPLE-powerExtrapolation Length=40962022.05 | 5.25 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 5.25 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.05 | 5.25 | |
| ALiBiL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 5.25 | |
| RotaryL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 5.25 | |
| KERPLE-powerExtrapolation Length=20482022.05 | 5.26 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.05 | 5.26 | |
| SandwichL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 5.27 | |
| SandwichL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 5.28 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.05 | 5.31 | |
| T5Extrapolation Length=40962022.05 | 5.32 | |
| SmoothedL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 5.33 | |
| SmoothedL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 5.33 | |
| ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.05 | 5.35 | |
| T5L_ex=4096, Training length=512, Embedding Type=learnable2022.12 | 5.35 | |
| ALiBiL_ex=1024, Training length=512, Embedding Type=parameter-free2022.12 | 5.41 | |
| SmoothedL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 5.45 | |
| T5Extrapolation Length=81922022.05 | 5.54 | |
| KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 5.56 | |
| T5Extrapolation Length=10242022.05 | 5.58 | |
| ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 5.58 | |
| ALiBiL_ex=2048, Training length=512, Embedding Type=parameter-free2022.12 | 5.58 | |
| ALiBiL_ex=4096, Training length=512, Embedding Type=parameter-free2022.12 | 5.58 | |
| ALiBiL_ex=8192, Training length=512, Embedding Type=parameter-free2022.12 | 5.58 | |
| KERPLE-logExtrapolation Length=10242022.05 | 5.61 | |
| T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.05 | 5.62 | |
| KERPLE-powerExtrapolation Length=10242022.05 | 5.65 | |
| ALiBiExtrapolation Length=20482022.05 | 5.71 | |
| ALiBiExtrapolation Length=81922022.05 | 5.74 | |
| ALiBiExtrapolation Length=163842022.05 | 5.78 | |
| SinusoidalL_ex=512, Training length=512, Embedding Type=parameter-free2022.12 | 5.8 | |
| ALiBiExtrapolation Length=10242022.05 | 5.82 | |
| ALiBiExtrapolation Length=40962022.05 | 5.87 | |
| T5Extrapolation Length=5122022.05 | 6.03 | |
| KERPLE-logExtrapolation Length=5122022.05 | 6.07 | |
| RotaryExtrapolation Length=5122022.05 | 6.07 | |
| KERPLE-powerExtrapolation Length=5122022.05 | 6.1 | |
| ALiBiExtrapolation Length=5122022.05 | 6.12 | |
| T5Extrapolation Length=163842022.05 | 6.25 | |
| T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.05 | 6.55 | |
| T5L_ex=8192, Training length=512, Embedding Type=learnable2022.12 | 6.74 | |
| FOTContext Length=64K, Zero-shot=true2023.07 | 6.81 | |
| FOTContext Length=16K, Zero-shot=true2023.07 | 6.94 | |
| FOTContext Length=4K, Zero-shot=true2023.07 | 7.44 |