Language modeling on PG-19 (val)
18.43Perplexity1x1-DenseFormer
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| 1x1-DenseFormerDepth=48, Dilation=1, DWA Period=1, Batch size=128, Training steps=48k2024.02 | 18.43 | — | — | |
| TransformerDepth=72, Batch size=128, Training steps=48k2024.02 | 18.44 | — | — | |
| TransformerDepth=48, Batch size=128, Training steps=48k2024.02 | 18.94 | — | — | |
| 1x1-DenseFormerDepth=24, Dilation=1, DWA Period=1, Batch size=128, Training steps=48k2024.02 | 19.6 | — | — | |
| TransformerDepth=24, Batch size=128, Training steps=48k2024.02 | 20.13 | — | — | |
| MEGALODON#Param.=1.3B2024.04 | 29.5 | — | — | |
| MEGABYTETokenizer=Bytes, Vocab Size=256, Context Length=8192 (bytes)2023.05 | 42.8 | — | — | |
| MEGABYTE#Param.=1.3B2024.04 | 42.8 | — | — | |
| Compressive TransformerContext length=512+512+2x512, Number of layers=362022.02 | 43.4 | — | — | |
| Compressive TransformerTokenizer=SentencePiece, Vocab Size=32k, Context Length=512+512+2x512 (subwords)2023.05 | 43.4 | — | — | |
| Compressive Trans.#Param.=-2024.04 | 43.4 | — | — | |
| Transformer-XLContext length=512+1024, Number of layers=362022.02 | 45.5 | — | — | |
| TransformerXLTokenizer=SentencePiece, Vocab Size=32k, Context Length=512+1024 (subwords)2023.05 | 45.5 | — | — | |
| Perceiver ARContext length=2048, Number of layers=602022.02 | 45.9 | — | — | |
| Perceiver ARContext length=4096, Number of layers=602022.02 | 45.9 | — | — | |
| PerceiverARTokenizer=SentencePiece, Vocab Size=32k, Context Length=2048 (subwords)2023.05 | 45.9 | — | — | |
| Perceiver AR#Param.=975M2024.04 | 45.9 | — | — | |
| PCAF-semanticSeq. (T)=1024, Seeds=1, Params=303.27M, Throughput=0.89M, Batch Size=2562026.06 | 50.94 | — | 26.13 | |
| PCAF-semanticSeq. (T)=2048, Seeds=1, Params=303.27M, Throughput=0.62M / 0.89M, Batch Size=32 / 2562026.06 | 52.45 | — | 27.69 | |
| Dense TransformerSeq. (T)=2048, Seeds=1, Params=302.77M, Throughput=0.43M, Batch Size=322026.06 | 53.84 | — | 29.39 | |
| Local TransformerSeq. (T)=2048, Seeds=1, Params=302.77M, Throughput=0.43M, Batch Size=322026.06 | 54.54 | — | 29.47 | |
| Dense TransformerSeq. (T)=1024, Seeds=1, Params=301.85M, Throughput=0.44M, Batch Size=322026.06 | 56.3 | — | 29.17 | |
| PCAF-contextSeq. (T)=2048, Seeds=1, Params=303.27M, Throughput=0.65M / 0.96M, Batch Size=32 / 2562026.06 | 57 | — | 27.91 | |
| Local TransformerSeq. (T)=1024, Seeds=1, Params=301.85M, Throughput=0.44M, Batch Size=322026.06 | 57.32 | — | 29.06 | |
| PCAF-contextSeq. (T)=1024, Seeds=1, Params=303.27M, Throughput=0.96M, Batch Size=2562026.06 | 59.31 | — | 26.64 | |
| Local convSeq. (T)=2048, Seeds=1, Params=303.27M, Throughput=0.78M / 1.20M, Batch Size=32 / 2562026.06 | 61.99 | — | 28.44 | |
| Local convSeq. (T)=1024, Seeds=1, Params=303.27M, Throughput=1.20M, Batch Size=2562026.06 | 63.79 | — | 27.03 | |
| Transformer byte-levelTokenizer=Bytes, Vocab Size=256, Context Length=2048 (bytes)2023.05 | 81.6 | — | — | |
| PerceiverAR byte-levelTokenizer=Bytes, Vocab Size=256, Context Length=8192 (bytes)2023.05 | 119.1 | — | — | |
| BaselineModel Size=370M2025.12 | — | 3.03 | — | |
| BaselineModel Size=1.3B2025.12 | — | 2.843 | — | |
| Dense-MoEModel Size=370M2025.12 | — | 3.062 | — | |
| FlashMHFModel Size=370M, Head Dimension (dh)=642025.12 | — | 3.046 | — | |
| FlashMHFModel Size=370M, Head Dimension (dh)=1282025.12 | — | 3.014 | — | |
| FlashMHFModel Size=370M, Head Dimension (dh)=2562025.12 | — | 3.029 | — | |
| FlashMHFModel Size=1.3B, Head Dimension (dh)=642025.12 | — | 2.849 | — | |
| FlashMHFModel Size=1.3B, Head Dimension (dh)=1282025.12 | — | 2.793 | — | |
| FlashMHFModel Size=1.3B, Head Dimension (dh)=2562025.12 | — | 2.799 | — | |
| KV-Foldpeak GPU=35.6 GB, total wall=166 s, T (Context Length)=128K, chain depth=511, Backbone=Llama-3.1-8B-Instruct2026.05 | — | 2.46 | — | |
| MH-FFNModel Size=370M, Head Dimension (dh)=1282025.12 | — | 3.031 | — | |
| PKVModel Size=370M, Head Dimension (dh)=1282025.12 | — | 3.334 | — | |
| StreamingLLMpeak GPU=16.6 GB, total wall=22 s, T (Context Length)=128K, chain depth=511, Backbone=Llama-3.1-8B-Instruct2026.05 | — | 2.66 | — |