Language Modeling on PG-19 (test)
9.765PerplexityERMAR
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ERMARContext Length=32k, Hardware=NVIDIA L40S GPU2025.03 | 9.765 | — | — | — | — | — | — | — | — | — | — | 90 | |
| MemLongContext Length=32k, Hardware=NVIDIA L40S GPU2025.03 | 9.858 | — | — | — | — | — | — | — | — | — | — | — | |
| BST:SH:UNSTRUCT-Leval seq. length=4096, window length=512, number params=371M, TPUv4 hours (k)=0.82023.06 | 10.37 | — | — | — | — | — | — | — | — | — | — | — | |
| BST:MF:UNSTRUCT-Leval seq. length=4096, window length=512, number params=388M, TPUv4 hours (k)=0.82023.06 | 10.42 | — | — | — | — | — | — | — | — | — | — | — | |
| BST:SH:S4-Leval seq. length=4096, window length=512, number params=366M, TPUv4 hours (k)=0.82023.06 | 10.47 | — | — | — | — | — | — | — | — | — | — | — | |
| GSS-HYBRID-Leval seq. length=4096, window length=512, number params=373M, TPUv4 hours (k)=0.82023.06 | 10.52 | — | — | — | — | — | — | — | — | — | — | — | |
| BST:MF:S4-Leval seq. length=4096, window length=512, number params=383M, TPUv4 hours (k)=0.82023.06 | 10.52 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=32k2025.02 | 10.91 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=20k2025.02 | 10.97 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=24k2025.02 | 11.01 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=28k2025.02 | 11.04 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=32k2025.02 | 11.05 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=20k2025.02 | 11.06 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=16k2025.02 | 11.07 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=20k2025.02 | 11.09 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=24k2025.02 | 11.1 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=28k2025.02 | 11.13 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=24k2025.02 | 11.14 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=16k2025.02 | 11.17 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=12k2025.02 | 11.18 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=16k2025.02 | 11.18 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=32k2025.02 | 11.18 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=28k2025.02 | 11.18 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=12k2025.02 | 11.25 | — | — | — | — | — | — | — | — | — | — | — | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target context window size=16k2025.02 | 11.27 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=12k2025.02 | 11.3 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=8k2025.02 | 11.32 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=8k2025.02 | 11.35 | — | — | — | — | — | — | — | — | — | — | — | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target context window size=12k2025.02 | 11.39 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=16k2025.02 | 11.39 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=8k2025.02 | 11.48 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=12k2025.02 | 11.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Dyn-NTKBackbone=Llama3-8b-ins-8k, Target context window size=1k2025.02 | 11.51 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=1k2025.02 | 11.52 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=1k2025.02 | 11.52 | — | — | — | — | — | — | — | — | — | — | — | |
| Dyn-NTKBackbone=Llama3-8b-ins-8k, Target context window size=4k2025.02 | 11.53 | — | — | — | — | — | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target context window size=4k2025.02 | 11.54 | — | — | — | — | — | — | — | — | — | — | — | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target context window size=8k2025.02 | 11.54 | — | — | — | — | — | — | — | — | — | — | — | |
| GALIBackbone=Llama3-8b-ins-8k, Target context window size=4k2025.02 | 11.54 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=8k2025.02 | 11.67 | — | — | — | — | — | — | — | — | — | — | — | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target context window size=1k2025.02 | 11.72 | — | — | — | — | — | — | — | — | — | — | — | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target context window size=4k2025.02 | 11.77 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=4k2025.02 | 11.81 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=1k2025.02 | 11.93 | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNBackbone=Llama3-8b-ins-8k, Target context window size=1k2025.02 | 11.93 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=4k2025.02 | 11.94 | — | — | — | — | — | — | — | — | — | — | — | |
| TRSF-XL:2048eval seq. length=2048, window length=2048, number params=190M, TPUv4 hours (k)=0.82023.06 | 11.96 | — | — | — | — | — | — | — | — | — | — | — | |
| SLIDE: 12Leval seq. length=4096, window length=512, number params=190M, TPUv4 hours (k)=0.52023.06 | 12.12 | — | — | — | — | — | — | — | — | — | — | — | |
| Dyn-NTKBackbone=Llama3-8b-ins-8k, Target context window size=8k2025.02 | 12.75 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=20k2025.02 | 13.03 | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid H3Number of Parameters=125M2022.12 | 16.2 | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerNumber of Parameters=125M2022.12 | 17 | — | — | — | — | — | — | — | — | — | — | — | |
| ECP TransformerNumber of Layers=24L, Parameters=214M2024.12 | 18.83 | — | — | — | — | — | — | — | — | — | — | — | |
| Linear AttentionNumber of Parameters=125M2022.12 | 19.1 | — | — | — | — | — | — | — | — | — | — | — | |
| FOTContext Length=64K, Zero-shot=true2023.07 | 22.65 | — | — | — | — | — | — | — | — | — | — | — | |
| FOTContext Length=16K, Zero-shot=true2023.07 | 22.85 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=24k2025.02 | 23 | — | — | — | — | — | — | — | — | — | — | — | |
| Memorizing TransformerContext Length=64K, Zero-shot=true2023.07 | 23.24 | — | — | — | — | — | — | — | — | — | — | — | |
| FOTContext Length=4K, Zero-shot=true2023.07 | 23.25 | — | — | — | — | — | — | — | — | — | — | — | |
| Memorizing TransformerContext Length=16K, Zero-shot=true2023.07 | 23.32 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XLContext Length=2K, Zero-shot=true2023.07 | 23.57 | — | — | — | — | — | — | — | — | — | — | — | |
| Memorizing TransformerContext Length=4K, Zero-shot=true2023.07 | 23.62 | — | — | — | — | — | — | — | — | — | — | — | |
| FOTContext Length=2K, Zero-shot=true2023.07 | 23.74 | — | — | — | — | — | — | — | — | — | — | — | |
| Memorizing TransformerContext Length=2K, Zero-shot=true2023.07 | 24.03 | — | — | — | — | — | — | — | — | — | — | — | |
| MEGALODON#Param.=1.3B2024.04 | 25.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Block-Recurrent TransformerLayers=24, parameters=1.3B, vocabulary size=32k2022.03 | 26.5 | — | — | — | — | — | — | — | — | — | — | — | |
| BlockRecurrentTokenizer=SentencePiece, Vocab Size=32k, Context Length=1024+recurrence (subwords)2023.05 | 26.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Block-Recurrent Trans.#Param.=1.3B2024.04 | 26.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Block Recurrent TransformerNumber of Layers=24L, Parameters=1.3B2024.12 | 26.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Block-Recurrent TransformerLayers=24, parameters=650M, vocabulary size=32k2022.03 | 28.46 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARContext length=2048, Number of layers=602022.02 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARLayers=60, parameters=974.6M, vocabulary size=32k2022.03 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | |
| PerceiverARTokenizer=SentencePiece, Vocab Size=32k, Context Length=2048 (subwords)2023.05 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver AR#Param.=975M2024.04 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARContext length=4096, Number of layers=602022.02 | 29 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-2 smallARACH=true, Logit offset (b)=-0.52026.03 | 33.11 | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerLayers=22, Heads=82020.03 | 33.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerContext length=8192, Number of layers=222022.02 | 33.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerLayers=22, parameters=490M, vocabulary size=98k2022.03 | 33.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerNumber of Layers=22L, Parameters=490M2024.12 | 33.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive TransformerLayers=362020.03 | 33.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive TransformerContext length=512+512+2x512, Number of layers=362022.02 | 33.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive TransformerLayers=36, vocabulary size=32k2022.03 | 33.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive TransformerTokenizer=SentencePiece, Vocab Size=32k, Context Length=512+512+2x512 (subwords)2023.05 | 33.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive Trans.#Param.=-2024.04 | 33.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive TransformerNumber of Layers=36L, Parameters=unknown2024.12 | 33.6 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-2 smallARACH=true, Logit offset (b)=-0.42026.03 | 33.82 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-2 smallARACH=true, Logit offset (b)=-0.32026.03 | 34.6 | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerXLLayers=362020.03 | 36.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XLContext length=512+1024, Number of layers=362022.02 | 36.3 | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerXLTokenizer=SentencePiece, Vocab Size=32k, Context Length=512+1024 (subwords)2023.05 | 36.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XLNumber of Layers=36L, Parameters=unknown2024.12 | 36.3 | — | — | — | — | — | — | — | — | — | — | — | |
| MEGABYTETokenizer=Bytes, Vocab Size=256, Context Length=8192 (bytes)2023.05 | 36.4 | — | — | — | — | — | — | — | — | — | — | — | |
| MEGABYTE#Param.=1.3B2024.04 | 36.4 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-2 smallARACH=false2026.03 | 37.33 | — | — | — | — | — | — | — | — | — | — | — | |
| Local TransformerLayers=24, Heads=82020.03 | 39.3 | — | — | — | — | — | — | — | — | — | — | — | |
| NTKBackbone=Llama3-8b-ins-8k, Target context window size=28k2025.02 | 42.95 | — | — | — | — | — | — | — | — | — | — | — | |
| adaptive kNN-LMn-gram overlap filtering=false2022.10 | 43.58 | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LMn-gram overlap filtering=false2022.10 | 43.93 | — | — | — | — | — | — | — | — | — | — | — | |
| adaptive kNN-LMn-gram overlap filtering=true2022.10 | 44.78 | — | — | — | — | — | — | — | — | — | — | — |