Language Modeling on GitHub (val)
1.83PerplexityBST:MF:UNSTRUCT-L
Evaluation Results
| Method | Links | |
|---|---|---|
| BST:MF:UNSTRUCT-Leval seq. length=4096, window length=512, number params=388M, TPUv4 hours (k)=1.82023.06 | 1.83 | |
| BST:MF:S4-Leval seq. length=4096, window length=512, number params=383M, TPUv4 hours (k)=1.82023.06 | 1.84 | |
| BST:SH:UNSTRUCT-Leval seq. length=4096, window length=512, number params=371M, TPUv4 hours (k)=1.82023.06 | 1.85 | |
| BST:SH:S4-Leval seq. length=4096, window length=512, number params=366M, TPUv4 hours (k)=1.82023.06 | 1.86 | |
| GSS-HYBRID-Leval seq. length=4096, window length=512, number params=373M, TPUv4 hours (k)=1.82023.06 | 1.88 | |
| TRSF-XL:2048eval seq. length=2048, window length=2048, number params=190M, TPUv4 hours (k)=32023.06 | 2.01 | |
| BST:MF:UNSTRUCTeval seq. length=4096, window length=512, number params=221M, TPUv4 hours (k)=1.82023.06 | 2.03 | |
| BRECT:FIXED:SKIPeval seq. length=4096, window length=512, number params=196M, TPUv4 hours (k)=32023.06 | 2.04 | |
| BST:MF:S4eval seq. length=4096, window length=512, number params=217M, TPUv4 hours (k)=1.82023.06 | 2.07 | |
| BST:SH:UNSTRUCTeval seq. length=4096, window length=512, number params=206M, TPUv4 hours (k)=1.82023.06 | 2.09 | |
| BST:SH:S4eval seq. length=4096, window length=512, number params=202M, TPUv4 hours (k)=1.82023.06 | 2.14 | |
| BST:MH:S4eval seq. length=4096, window length=512, number params=218M, TPUv4 hours (k)=1.82023.06 | 2.15 | |
| SLIDE: 12Leval seq. length=4096, window length=512, number params=190M, TPUv4 hours (k)=1.82023.06 | 2.28 |