Language Modeling on OpenWebText
5.33PerplexityDFM
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DFMStep=1, Distillation Method=ESD2026.07 | 5.33 | — | — | — | 0.26 | — | — | — | |
| Hybrid H3Number of Parameters=2.7B2022.12 | 11 | — | — | — | — | — | — | — | |
| GPT-NeoNumber of Parameters=2.7B2022.12 | 11.7 | — | — | — | — | — | — | — | |
| Hybrid H3Number of Parameters=1.3B2022.12 | 12.4 | — | — | — | — | — | — | — | |
| Original ModelBackbone=Llama-3-8B, Avg. Sparsity (L/H)=0% / 0%, Avg. Mem (GB)=4.50, Time (s)=34522026.06 | 12.45 | — | — | — | — | — | — | — | |
| Oracle Static PruningBackbone=Llama-3-8B, Avg. Sparsity (L/H)=33% / 20%, Avg. Mem (GB)=3.80, Time (s)=22502026.06 | 12.55 | — | — | — | — | — | — | — | |
| Learning to AllocateBackbone=Llama-3-8B, Avg. Sparsity (L/H)=34% / 24%, Avg. Mem (GB)=4.10, Time (s)=23802026.06 | 12.58 | — | — | — | — | — | — | — | |
| FlexiDepthBackbone=Llama-3-8B, Avg. Sparsity (L/H)=32% / –, Avg. Mem (GB)=4.05, Time (s)=24202026.06 | 12.6 | — | — | — | — | — | — | — | |
| AdaSkipBackbone=Llama-3-8B, Avg. Sparsity (L/H)=35% / –, Avg. Mem (GB)=3.88, Time (s)=23502026.06 | 12.65 | — | — | — | — | — | — | — | |
| GPT-2 XLNumber of Parameters=1.5B2022.12 | 12.9 | — | — | — | — | — | — | — | |
| GPT-NeoNumber of Parameters=1.3B2022.12 | 13.1 | — | — | — | — | — | — | — | |
| Static PruningBackbone=Llama-3-8B, Avg. Sparsity (L/H)=33% / 0%, Avg. Mem (GB)=3.92, Time (s)=23102026.06 | 13.8 | — | — | — | — | — | — | — | |
| ConceptLMModel Size=1.5B2026.02 | 14.4 | — | 26.38 | — | — | — | — | — | |
| ContextLMModel Size=1.5B2026.02 | 14.6 | — | 25.93 | — | — | — | — | — | |
| GPT2-PMModel Size=1.5B2026.02 | 14.94 | — | 30.01 | — | — | — | — | — | |
| GPT2-1.5BModel Size=1.5B2026.02 | 14.99 | — | 29.68 | — | — | — | — | — | |
| Dense (full)Model=GPT2-350M, Latency (128k context)=46.782026.03 | 15.03 | — | — | — | — | — | — | — | |
| ConceptLMModel Size=774M2026.02 | 15.2 | — | 29.61 | — | — | — | — | — | |
| ContextLMModel Size=774M2026.02 | 15.41 | — | 31.8 | — | — | — | — | — | |
| GPT2-PMModel Size=774M2026.02 | 15.89 | — | 32.89 | — | — | — | — | — | |
| Hybrid H3Number of Parameters=355M2022.12 | 15.9 | — | — | — | — | — | — | — | |
| GPT2-774MModel Size=774M2026.02 | 16.01 | — | 36.35 | — | — | — | — | — | |
| Uni-EDLM-NCEvariant=NCE, window size w=0.2, candidate size k=22026.06 | 16.54 | — | — | — | — | — | — | — | |
| SFA (k = 8)Model=GPT2-350M, Latency (128k context)=23.672026.03 | 16.78 | — | — | — | — | — | — | — | |
| ConceptLMModel Size=355M2026.02 | 16.81 | — | 40.51 | — | — | — | — | — | |
| GPT-2 mediumNumber of Parameters=355M2022.12 | 17 | — | — | — | — | — | — | — | |
| ContextLMModel Size=355M2026.02 | 17.03 | — | 43.06 | — | — | — | — | — | |
| Dense (full)Model=GPT2-124M, Latency (128k context)=16.862026.03 | 17.29 | — | — | — | — | — | — | — | |
| Dense (full)Model=GPT2 124M, Operation Category=Baseline, Latency@128k (Decode)=17.08, Latency@128k (Forward)=16.862026.03 | 17.29 | — | — | — | — | — | — | — | |
| MLAModel=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=5.04, Latency@128k (Forward)=15.392026.03 | 17.38 | — | — | — | — | — | — | — | |
| Uni-EDLM-ARvariant=AR, window size w=0.2, candidate size k=22026.06 | 17.41 | — | — | — | — | — | — | — | |
| Autoregressive TransformerSteps=1M2026.05 | 17.5 | — | — | — | — | — | — | — | |
| AR†Architecture=Transformer (GPT-2-small)2026.06 | 17.56 | — | — | — | — | — | — | — | |
| EDLM-CoAR†Type=Dependency-aware DLM, variant=CoAR2026.06 | 17.58 | — | — | — | — | — | — | — | |
| CARDzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 17.59 | — | — | — | — | — | — | — | |
| QuantModel=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=14.26, Latency@128k (Forward)=12.972026.03 | 17.64 | — | — | — | — | — | — | — | |
| GPT2-PMModel Size=355M2026.02 | 17.67 | — | 52.55 | — | — | — | — | — | |
| ARMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 17.68 | — | — | — | — | — | — | — | |
| Autoregressive TransformerSteps=250K2026.05 | 17.8 | — | — | — | — | — | — | — | |
| GPT2-355MModel Size=355M2026.02 | 17.97 | — | 52.57 | — | — | — | — | — | |
| SFA (k = 8)Model=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=14.12, Latency@128k (Forward)=9.412026.03 | 18.17 | — | — | — | — | — | — | — | |
| SFA (k = 8)Model=GPT2-124M, Latency (128k context)=9.412026.03 | 18.27 | — | — | — | — | — | — | — | |
| SFA (quant)Model=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=12.28, Latency@128k (Forward)=8.722026.03 | 18.54 | — | — | — | — | — | — | — | |
| LongformerModel=GPT2 124M, Operation Category=Token-Level Operation, Latency@128k (Decode)=6.75, Latency@128k (Forward)=7.932026.03 | 18.73 | — | — | — | — | — | — | — | |
| MLA + SFAModel=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=3.98, Latency@128k (Forward)=15.052026.03 | 19.07 | — | — | — | — | — | — | — | |
| +SFA (k = 8)Model=GPT2 124M, Operation Category=Token-Level Operation, Latency@128k (Decode)=5.23, Latency@128k (Forward)=6.182026.03 | 19.3 | — | — | — | — | — | — | — | |
| Dense (d = 32)Model=GPT2-350M, Latency (128k context)=20.582026.03 | 19.89 | — | — | — | — | — | — | — | |
| Low-RankModel=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=8.93, Latency@128k (Forward)=7.992026.03 | 19.89 | — | — | — | — | — | — | — | |
| RFMoEScale=L, Size=870.6M, FLOPs=613.2M2026.04 | 19.97 | — | — | — | — | — | — | — | |
| GPT-2 (dense attention)Setting=Fine-tuned2026.05 | 20.15 | — | — | — | — | — | — | — | |
| MoSEBackbone Model=GPT2-STANDARD (322M), Training Budget=15B tokens, Execution Mode=Test-time training (TTT), Efficient FLOPs=true2026.02 | 20.38 | — | — | — | — | — | — | — | |
| ConceptLMModel Size=124M2026.02 | 20.56 | — | 84.75 | — | — | — | — | — | |
| MoSEBackbone Model=GPT2-STANDARD (322M), Training Budget=15B tokens, Execution Mode=Uniform-width execution (w=1.0)2026.02 | 20.65 | — | — | — | — | — | — | — | |
| ContextLMModel Size=124M2026.02 | 20.68 | — | 87.19 | — | — | — | — | — | |
| Short (d = 32)Model=GPT2 124M, Operation Category=Feature-Level Operation, Latency@128k (Decode)=8.37, Latency@128k (Forward)=7.862026.03 | 20.7 | — | — | — | — | — | — | — | |
| MDLMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 20.77 | — | — | — | — | — | — | — | |
| MoEBackbone Model=GPT2-STANDARD (322M), Training Budget=15B tokens2026.02 | 20.81 | — | — | — | — | — | — | — | |
| Dense (d = 32)Model=GPT2-124M, Latency (128k context)=7.862026.03 | 20.88 | — | — | — | — | — | — | — | |
| Hybrid H3Number of Parameters=125M2022.12 | 20.9 | — | — | — | — | — | — | — | |
| ARcontext length=1024, sampling strategy=top-500, reference model=GPT-2 large2024.12 | 20.98 | — | — | — | — | — | — | — | |
| GPT2-PMModel Size=124M2026.02 | 21.22 | — | 99.14 | — | — | — | — | — | |
| Block Diffusion*Type=Traditional DLM2026.06 | 21.27 | — | — | — | — | — | — | — | |
| GPT-2 + ChaCALSetting=Fine-tuned2026.05 | 21.46 | — | — | — | — | — | — | — | |
| EDLM-NCE†Type=Dependency-aware DLM, variant=NCE2026.06 | 21.52 | — | — | — | — | — | — | — | |
| GPT2-124MModel Size=124M2026.02 | 22.04 | — | 101.94 | — | — | — | — | — | |
| RFMoEScale=M, Size=307.3M, FLOPs=249.2M2026.04 | 22.08 | — | — | — | — | — | — | — | |
| Block-ChaCALSetting=Fine-tuned, block size=m=n//32026.05 | 22.08 | — | — | — | — | — | — | — | |
| REPLAIDSteps=1M, s.c.=true2026.05 | 22.1 | — | — | — | — | — | — | — | |
| HTMuonModel=GPT-2 small2026.03 | 22.2 | — | — | — | — | — | — | — | |
| DenseSparsity=0%, Model #param=163M, Extra #param=02023.05 | 22.4 | — | — | — | — | — | — | — | |
| MuonModel=GPT-2 small2026.03 | 22.46 | — | — | — | — | — | — | — | |
| GPT-2 smallNumber of Parameters=125M2022.12 | 22.6 | — | — | — | — | — | — | — | |
| GPT-NeoNumber of Parameters=125M2022.12 | 22.6 | — | — | — | — | — | — | — | |
| Block-ChaCALSetting=Fine-tuned, block size=m=n//22026.05 | 22.6 | — | — | — | — | — | — | — | |
| Block-ChaCALSetting=Fine-tuned, block size=m=n//42026.05 | 22.6 | — | — | — | — | — | — | — | |
| MDLM (low var.)Steps=1M, Retrained from scratch=true2026.05 | 23.1 | — | — | — | — | — | — | — | |
| MDLMSteps=1M2026.05 | 23.2 | — | — | — | — | — | — | — | |
| Duo‡Type=Traditional DLM2026.06 | 23.25 | — | — | — | — | — | — | — | |
| DCD*Type=Dependency-aware DLM2026.06 | 23.33 | — | — | — | — | — | — | — | |
| REPLAIDSteps=1M, s.c.=false2026.05 | 23.6 | — | — | — | — | — | — | — | |
| GPT-2 (pre-trained)Setting=Pretrained2026.05 | 23.78 | — | — | — | — | — | — | — | |
| MDLM†Type=Traditional DLM2026.06 | 23.83 | — | — | — | — | — | — | — | |
| PlaidSteps=1M, s.c.=true2026.05 | 24.4 | — | — | — | — | — | — | — | |
| SEDD†Type=Traditional DLM2026.06 | 24.56 | — | — | — | — | — | — | — | |
| MoEScale=L, Size=808.4M, FLOPs=608.4M2026.04 | 24.58 | — | — | — | — | — | — | — | |
| REPLAIDSteps=250K, s.c.=true2026.05 | 24.9 | — | — | — | — | — | — | — | |
| MoEScale=M, Size=289.9M, FLOPs=248.0M2026.04 | 25 | — | — | — | — | — | — | — | |
| MDLM (low var.)Steps=250K, Retrained from scratch=true2026.05 | 25 | — | — | — | — | — | — | — | |
| AdamWModel=GPT-2 small2026.03 | 25.19 | — | — | — | — | — | — | — | |
| MDLMSteps=250K2026.05 | 25.2 | — | — | — | — | — | — | — | |
| DuoSteps=1M2026.05 | 25.2 | — | — | — | — | — | — | — | |
| PlaidSteps=250K, s.c.=true2026.05 | 25.4 | — | — | — | — | — | — | — | |
| PlaidSteps=1M, s.c.=false2026.05 | 25.7 | — | — | — | — | — | — | — | |
| SEDD AbsorbSteps=250K2026.05 | 26.8 | — | — | — | — | — | — | — | |
| REPLAIDSteps=250K, s.c.=false2026.05 | 26.8 | — | — | — | — | — | — | — | |
| DuoSteps=250K2026.05 | 27.1 | — | — | — | — | — | — | — | |
| MoSEBackbone Model=GPT2-STANDARD (322M), Training Budget=3B tokens, Execution Mode=Test-time training (TTT), Efficient FLOPs=true2026.02 | 27.32 | — | — | — | — | — | — | — | |
| UDLMSteps=1M2026.05 | 27.4 | — | — | — | — | — | — | — | |
| RFMoEScale=S, Size=95.32M, FLOPs=91.08M2026.04 | 27.42 | — | — | — | — | — | — | — | |
| PlaidSteps=250K, s.c.=false2026.05 | 27.5 | — | — | — | — | — | — | — |