List your company
ResearchFormerly WizWand

Language Modeling on The Pile (val)

0.62Perplexity (bits/byte)No intervention
-0.07764.63129.3414.0488Jun 6, 2023Dec 10, 2023Jun 15, 2024Dec 20, 2024Jun 26, 2025Dec 31, 2025Jul 7, 2026
Updated 3mo ago

Evaluation results

MethodLinks
No interventionModel Family=Pythia, Model Size=12B2023.060.62
Random erasureModel Family=Pythia, Model Size=12B2023.060.63
No interventionModel Family=Pythia, Model Size=6.9B2023.060.64
Random erasureModel Family=Pythia, Model Size=6.9B2023.060.66
No interventionModel Family=Pythia, Model Size=1.4B2023.060.7
Random erasureModel Family=Pythia, Model Size=1.4B2023.060.72
No interventionModel Family=Pythia, Model Size=160M2023.060.9
Random erasureModel Family=Pythia, Model Size=160M2023.060.99
LEACEModel Family=Pythia, Model Size=1.4B2023.062.25
unigram entropyModel Family=Pythia, Model Size=160M2023.062.66
unigram entropyModel Family=Pythia, Model Size=1.4B2023.062.66
unigram entropyModel Family=Pythia, Model Size=6.9B2023.062.66
unigram entropyModel Family=Pythia, Model Size=12B2023.062.66
LEACEModel Family=Pythia, Model Size=160M2023.062.79
LEACEModel Family=Pythia, Model Size=12B2023.063.2
SALModel Family=Pythia, Model Size=1.4B2023.063.44
SALModel Family=Pythia, Model Size=160M2023.063.53
LEACEModel Family=Pythia, Model Size=6.9B2023.063.57
SALModel Family=Pythia, Model Size=6.9B2023.064.17
SALModel Family=Pythia, Model Size=12B2023.064.69
AC-ODMModel size=7B, Training steps=20,832, Architecture=LLaMA-style, Execution mode=non-proxy2025.058.79
AC-ODMModel size=7B, Training steps=15,624, Architecture=LLaMA-style, Execution mode=non-proxy2025.059.54
TriRouteScale=1.3B, FLOPs=0.55, KV mem=0.402026.079.7
DenseScale=1.3B, FLOPs=1.00, KV mem=1.002026.079.8
Independent comboScale=1.3B, FLOPs=0.55, KV mem=0.402026.0710.1
MoD + KV-quantScale=1.3B, FLOPs=0.55, KV mem=0.402026.0710.5
AC-ODMModel size=3B, Training steps=20,832, Architecture=LLaMA-style, Execution mode=non-proxy2025.0510.59
AC-ODMModel size=7B, Training steps=10,416, Architecture=LLaMA-style, Execution mode=non-proxy2025.0510.99
AC-ODMModel size=3B, Training steps=15,624, Architecture=LLaMA-style, Execution mode=non-proxy2025.0511.15
DenseScale=410M, FLOPs=1.00, KV mem=1.002026.0711.6
TriRouteScale=410M, FLOPs=0.55, KV mem=0.402026.0711.7
Independent comboScale=410M, FLOPs=0.55, KV mem=0.402026.0712
MoD + KV-quantScale=410M, FLOPs=0.55, KV mem=0.402026.0712.3
AC-ODMModel size=0.9B, Training steps=20,832, Architecture=LLaMA-style, Execution mode=non-proxy2025.0512.86
MHA2026.0312.929
MLA2026.0312.965
MLRABranches=42026.0313.124
GLABranches=22026.0313.225
GLABranches=42026.0313.229
MLRABranches=22026.0313.236
AC-ODMModel size=3B, Training steps=10,416, Architecture=LLaMA-style, Execution mode=non-proxy2025.0513.24
TPA2026.0313.333
GTA2026.0313.402
AC-ODMModel size=0.9B, Training steps=15,624, Architecture=LLaMA-style, Execution mode=non-proxy2025.0513.65
GQA2026.0313.758
MFA2026.0313.903
AC-ODMModel size=7B, Training steps=5,208, Architecture=LLaMA-style, Execution mode=non-proxy2025.0513.98
MQA2026.0314.008
DenseScale=160M, FLOPs=1.00, KV mem=1.002026.0714.5
TriRouteScale=160M, FLOPs=0.55, KV mem=0.402026.0714.6
AC-ODMModel size=0.9B, Training steps=10,416, Architecture=LLaMA-style, Execution mode=non-proxy2025.0514.82
Independent comboScale=160M, FLOPs=0.55, KV mem=0.402026.0715
AC-ODMModel size=3B, Training steps=5,208, Architecture=LLaMA-style, Execution mode=non-proxy2025.0515.31
MoD + KV-quantScale=160M, FLOPs=0.55, KV mem=0.402026.0715.4
AC-ODMModel size=0.9B, Training steps=5,208, Architecture=LLaMA-style, Execution mode=non-proxy2025.0518.06