Language Modeling on WikiText (test)
4.88PerplexityFP16
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| FP16Model=LLaMA-2-13B, Quantization=None, Calibration=FP162026.04 | 4.88 | — | — | — | — | |
| COVERCALModel=LLaMA-2-13B, Quantization=AWQ INT4, Calibration Samples=128, Calibration=COVERCAL2026.04 | 4.94 | — | — | — | — | |
| Max-ActVarModel=LLaMA-2-13B, Quantization=AWQ INT4, Calibration Samples=128, Calibration=Max-ActVar2026.04 | 5.03 | — | — | — | — | |
| RandomModel=LLaMA-2-13B, Quantization=AWQ INT4, Calibration Samples=128, Calibration=Random2026.04 | 5.08 | — | — | — | — | |
| FullBackbone=Mistral-v2-7B, Sparsity=0%2025.12 | 5.49 | — | — | — | — | |
| TEALBackbone=Mistral-v2-7B, Sparsity=25%2025.12 | 5.52 | — | — | — | — | |
| SPONBackbone=Mistral-v2-7B, Sparsity=25%2025.12 | 5.58 | — | — | — | — | |
| SPONBackbone=Mistral-v2-7B, Sparsity=50%2025.12 | 5.86 | — | — | — | — | |
| TEALBackbone=Mistral-v2-7B, Sparsity=50%2025.12 | 6 | — | — | — | — | |
| WandaBackbone=Mistral-v2-7B, Sparsity=50%2025.12 | 6.26 | — | — | — | — | |
| SPONBackbone=Mistral-v2-7B, Sparsity=60%2025.12 | 6.51 | — | — | — | — | |
| FullBackbone=Llama3-8B, Sparsity=0%2025.12 | 6.75 | — | — | — | — | |
| TEALBackbone=Llama3-8B, Sparsity=25%2025.12 | 6.88 | — | — | — | — | |
| TEALBackbone=Mistral-v2-7B, Sparsity=60%2025.12 | 6.9 | — | — | — | — | |
| SPONBackbone=Llama3-8B, Sparsity=25%2025.12 | 6.92 | — | — | — | — | |
| SPONBackbone=Llama3-8B, Sparsity=50%2025.12 | 7.83 | — | — | — | — | |
| TEALBackbone=Llama3-8B, Sparsity=50%2025.12 | 8.34 | — | — | — | — | |
| SPONBackbone=Qwen3-8B, Sparsity=25%2025.12 | 8.94 | — | — | — | — | |
| FullBackbone=Qwen3-8B, Sparsity=0%2025.12 | 8.99 | — | — | — | — | |
| TEALBackbone=Qwen3-8B, Sparsity=25%2025.12 | 9.04 | — | — | — | — | |
| SPONBackbone=Qwen3-8B, Sparsity=50%2025.12 | 9.26 | — | — | — | — | |
| SPONBackbone=Llama3-8B, Sparsity=60%2025.12 | 9.63 | — | — | — | — | |
| WandaBackbone=Llama3-8B, Sparsity=50%2025.12 | 9.66 | — | — | — | — | |
| TEALBackbone=Qwen3-8B, Sparsity=50%2025.12 | 9.75 | — | — | — | — | |
| WandaBackbone=Qwen3-8B, Sparsity=50%2025.12 | 10.41 | — | — | — | — | |
| SPONBackbone=Qwen3-8B, Sparsity=60%2025.12 | 10.42 | — | — | — | — | |
| TEALBackbone=Qwen3-8B, Sparsity=60%2025.12 | 11.38 | — | — | — | — | |
| TEALBackbone=Llama3-8B, Sparsity=60%2025.12 | 11.62 | — | — | — | — | |
| FullBackbone=Llama3-1B, Sparsity=0%2025.12 | 12.14 | — | — | — | — | |
| SPONBackbone=Llama3-1B, Sparsity=50%2025.12 | 15.43 | — | — | — | — | |
| TEALBackbone=Llama3-1B, Sparsity=50%2025.12 | 17.03 | — | — | — | — | |
| GPT-2 (MLE)Model Size=Large, Objective=MLE, Sampling=Unbiased2023.05 | 18.24 | 90 | 0.75 | 56.7 | — | |
| GPT-2 (MLE)Model Size=Medium, Objective=MLE, Sampling=Unbiased2023.05 | 20.43 | 90 | 0.73 | 57.3 | — | |
| Subspace Networks (Decentralized Compressed)B/W=80Mbps, TPS=592.412025.06 | 23.01 | — | — | — | — | |
| CentralizedB/W=100Gbps, TPS=602.572025.06 | 23.08 | — | — | — | — | |
| GPT-2 (MIXCE)Model Size=Large, Objective=MIXCE, Sampling=Unbiased2023.05 | 23.44 | 88 | 0.95 | 57.8 | — | |
| ARzero-shot=true2024.10 | 25.75 | — | — | — | — | |
| GPT-2Zero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 25.75 | — | — | — | — | |
| GPT-2 (MIXCE)Model Size=Medium, Objective=MIXCE, Sampling=Unbiased2023.05 | 25.92 | 88 | 0.95 | 58.4 | — | |
| GPT-2 (MLE)Model Size=Small, Objective=MLE, Sampling=Unbiased2023.05 | 26.98 | 91 | 0.67 | 55.6 | — | |
| DenseFormerParameters=468M, Training Tokens=20B2024.10 | 28 | — | — | — | — | |
| EDLM-coARzero-shot=true2024.10 | 28.31 | — | — | — | — | |
| EDLM-ARzero-shot=true2024.10 | 29.24 | — | — | — | — | |
| WandaBackbone=Llama3-1B, Sparsity=50%2025.12 | 29.68 | — | — | — | — | |
| EDLM-NCEzero-shot=true2024.10 | 30.77 | — | — | — | — | |
| Learnable ResFormer plusParameters=468M, Training Tokens=20B2024.10 | 31.4 | — | — | — | — | |
| Learnable ResFormerParameters=468M, Training Tokens=20B2024.10 | 32.5 | — | — | — | — | |
| MLDMzero-shot=true2024.10 | 32.83 | — | — | — | — | |
| Identity ResFormerParameters=468M, Training Tokens=20B2024.10 | 32.9 | — | — | — | — | |
| LDDM-MDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText2025.10 | 33.27 | — | — | — | — | |
| TransformerParameters=468M, Training Tokens=20B2024.10 | 33.4 | — | — | — | — | |
| SEDDzero-shot=true2024.10 | 34.28 | — | — | — | — | |
| DonorArchitecture=GPT-OSS-20B, Trainable %=14.6%, Time=2.7m2026.01 | 34.56 | — | — | — | — | |
| VADDZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 34.78 | — | — | — | — | |
| GPT-2 (MIXCE)Model Size=Small, Objective=MIXCE, Sampling=Unbiased2023.05 | 35.04 | 87 | 0.93 | 56.7 | — | |
| RADD-AOZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 35.25 | — | — | — | — | |
| MDLMZero-shot=true, Trained on=OpenWebText, Training iterations=1M, Reproduced=true2025.05 | 35.61 | — | — | — | — | |
| NeuTRENOParameters=468M, Training Tokens=20B2024.10 | 36.3 | — | — | — | — | |
| MDLMDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.10 | 36.3 | — | — | — | — | |
| SEDD AbsorbDiffusion Framework=Masked Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.10 | 38.34 | — | — | — | — | |
| UDLMDiffusion Framework=Uniform Diffusion, Training Steps=1 million, Training Dataset=OpenWebText, Retrained=true2025.10 | 38.48 | — | — | — | — | |
| LDDM-UDiffusion Framework=Uniform Diffusion, Training Steps=1 million, Training Dataset=OpenWebText2025.10 | 38.89 | — | — | — | — | |
| SEDD-AbsorbZero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 40.62 | — | — | — | — | |
| Fixed Neuron PruningBackbone=Llama3-1B, Sparsity=50%2025.12 | 62.36 | — | — | — | — | |
| LoRAArchitecture=GPT-OSS-20B, Trainable %=0.01%, Time=76.2m2026.01 | 98.37 | — | — | — | — | |
| IA3Architecture=GPT-OSS-20B, Trainable %=0.0001%, Time=76.5m2026.01 | 214.99 | — | — | — | — | |
| Top-KArchitecture=GPT-OSS-20B, Trainable %=10.6%, Time=22.2m2026.01 | 365.83 | — | — | — | — | |
| Zero-shotArchitecture=GPT-OSS-20B, Trainable %=0%2026.01 | 397.78 | — | — | — | — | |
| DecentralizedB/W=80Mbps, TPS=36.122025.06 | 601.84 | — | — | — | — | |
| HumanSampling=Unbiased2023.05 | — | 89 | 1 | 62.8 | — | |
| Humandecoding=unbiased sampling2023.05 | — | 89 | 1 | 62.8 | — | |
| MIXCEModel Size=Small, best p=0.99, decoding=top-p sampling2023.05 | — | 87 | 0.95 | 56.8 | — | |
| MIXCEModel Size=Medium, best p=0.99, decoding=top-p sampling2023.05 | — | 87 | 0.96 | 59 | — | |
| MIXCEModel Size=Large, best p=0.99, decoding=top-p sampling2023.05 | — | 87 | 0.97 | 58 | — | |
| MLEModel Size=Small, best p=0.85, decoding=top-p sampling2023.05 | — | 89 | 0.93 | 58.4 | — | |
| MLEModel Size=Medium, best p=0.85, decoding=top-p sampling2023.05 | — | 88 | 0.95 | 60.2 | — | |
| MLEModel Size=Large, best p=0.87, decoding=top-p sampling2023.05 | — | 89 | 0.96 | 59.4 | — |