Language Modeling on arXiv tokens (test)
1.22Bits Per TokenFeedback:lstm:single
Evaluation Results
| Method | Links | |
|---|---|---|
| Feedback:lstm:singlesegment length=4096, window length=512, step time (relative)=1.42022.03 | 1.22 | |
| Memorizing Trans. 64ksegment length=512, window length=512, step time (relative)=1.942022.03 | 1.22 | |
| Rec:fixed:skipsegment length=4096, window length=512, step time (relative)=0.992022.03 | 1.24 | |
| Feedback:fixed:skipsegment length=4096, window length=512, step time (relative)=1.352022.03 | 1.24 | |
| Rec:fixed:singlesegment length=4096, window length=512, step time (relative)=1.022022.03 | 1.25 | |
| Rec:lstm:dualsegment length=4096, window length=512, step time (relative)=1.062022.03 | 1.26 | |
| Rec:fixed:dualsegment length=4096, window length=512, step time (relative)=1.012022.03 | 1.27 | |
| Rec:lstm:singlesegment length=4096, window length=512, step time (relative)=1.052022.03 | 1.29 | |
| XL:2048segment length=2048, window length=2048, step time (relative)=2.112022.03 | 1.31 | |
| Rec:lstm:skipsegment length=4096, window length=512, step time (relative)=12022.03 | 1.31 | |
| XL:1024segment length=1024, window length=1024, step time (relative)=1.22022.03 | 1.37 | |
| Slide:13Lsegment length=4096, window length=512, step time (relative)=12022.03 | 1.42 | |
| Slide:12Lsegment length=4096, window length=512, step time (relative)=0.932022.03 | 1.43 | |
| XL:512segment length=512, window length=512, step time (relative)=0.882022.03 | 1.45 |