Language Modeling on The Pile (val)
0.62Perplexity (bits/byte)No intervention
Evaluation Results
| Method | Links | |
|---|---|---|
| No interventionModel Family=Pythia, Model Size=12B2023.06 | 0.62 | |
| Random erasureModel Family=Pythia, Model Size=12B2023.06 | 0.63 | |
| No interventionModel Family=Pythia, Model Size=6.9B2023.06 | 0.64 | |
| Random erasureModel Family=Pythia, Model Size=6.9B2023.06 | 0.66 | |
| No interventionModel Family=Pythia, Model Size=1.4B2023.06 | 0.7 | |
| Random erasureModel Family=Pythia, Model Size=1.4B2023.06 | 0.72 | |
| No interventionModel Family=Pythia, Model Size=160M2023.06 | 0.9 | |
| Random erasureModel Family=Pythia, Model Size=160M2023.06 | 0.99 | |
| LEACEModel Family=Pythia, Model Size=1.4B2023.06 | 2.25 | |
| unigram entropyModel Family=Pythia, Model Size=160M2023.06 | 2.66 | |
| unigram entropyModel Family=Pythia, Model Size=1.4B2023.06 | 2.66 | |
| unigram entropyModel Family=Pythia, Model Size=6.9B2023.06 | 2.66 | |
| unigram entropyModel Family=Pythia, Model Size=12B2023.06 | 2.66 | |
| LEACEModel Family=Pythia, Model Size=160M2023.06 | 2.79 | |
| LEACEModel Family=Pythia, Model Size=12B2023.06 | 3.2 | |
| SALModel Family=Pythia, Model Size=1.4B2023.06 | 3.44 | |
| SALModel Family=Pythia, Model Size=160M2023.06 | 3.53 | |
| LEACEModel Family=Pythia, Model Size=6.9B2023.06 | 3.57 | |
| SALModel Family=Pythia, Model Size=6.9B2023.06 | 4.17 | |
| SALModel Family=Pythia, Model Size=12B2023.06 | 4.69 | |
| AC-ODMModel size=7B, Training steps=20,832, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 8.79 | |
| AC-ODMModel size=7B, Training steps=15,624, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 9.54 | |
| TriRouteScale=1.3B, FLOPs=0.55, KV mem=0.402026.07 | 9.7 | |
| DenseScale=1.3B, FLOPs=1.00, KV mem=1.002026.07 | 9.8 | |
| Independent comboScale=1.3B, FLOPs=0.55, KV mem=0.402026.07 | 10.1 | |
| MoD + KV-quantScale=1.3B, FLOPs=0.55, KV mem=0.402026.07 | 10.5 | |
| AC-ODMModel size=3B, Training steps=20,832, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 10.59 | |
| AC-ODMModel size=7B, Training steps=10,416, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 10.99 | |
| AC-ODMModel size=3B, Training steps=15,624, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 11.15 | |
| DenseScale=410M, FLOPs=1.00, KV mem=1.002026.07 | 11.6 | |
| TriRouteScale=410M, FLOPs=0.55, KV mem=0.402026.07 | 11.7 | |
| Independent comboScale=410M, FLOPs=0.55, KV mem=0.402026.07 | 12 | |
| MoD + KV-quantScale=410M, FLOPs=0.55, KV mem=0.402026.07 | 12.3 | |
| AC-ODMModel size=0.9B, Training steps=20,832, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 12.86 | |
| MHA2026.03 | 12.929 | |
| MLA2026.03 | 12.965 | |
| MLRABranches=42026.03 | 13.124 | |
| GLABranches=22026.03 | 13.225 | |
| GLABranches=42026.03 | 13.229 | |
| MLRABranches=22026.03 | 13.236 | |
| AC-ODMModel size=3B, Training steps=10,416, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 13.24 | |
| TPA2026.03 | 13.333 | |
| GTA2026.03 | 13.402 | |
| AC-ODMModel size=0.9B, Training steps=15,624, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 13.65 | |
| GQA2026.03 | 13.758 | |
| MFA2026.03 | 13.903 | |
| AC-ODMModel size=7B, Training steps=5,208, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 13.98 | |
| MQA2026.03 | 14.008 | |
| DenseScale=160M, FLOPs=1.00, KV mem=1.002026.07 | 14.5 | |
| TriRouteScale=160M, FLOPs=0.55, KV mem=0.402026.07 | 14.6 | |
| AC-ODMModel size=0.9B, Training steps=10,416, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 14.82 | |
| Independent comboScale=160M, FLOPs=0.55, KV mem=0.402026.07 | 15 | |
| AC-ODMModel size=3B, Training steps=5,208, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 15.31 | |
| MoD + KV-quantScale=160M, FLOPs=0.55, KV mem=0.402026.07 | 15.4 | |
| AC-ODMModel size=0.9B, Training steps=5,208, Architecture=LLaMA-style, Execution mode=non-proxy2025.05 | 18.06 |