Language Modeling on WikiText-103 (test)
2.22PerplexityRETRO
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RETRORetrieval Mode=On, Number of Parameters=7.5B2021.12 | 2.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RETRORetrieval Mode=On, Number of Parameters=172M2021.12 | 3.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADEPTModel Family=Llama, Model Size=65B2026.01 | 5.5 | — | — | — | 54.9 | — | — | — | — | — | — | — | — | — | |
| ADEPTModel Family=Llama, Model Size=33B2026.01 | 6.2 | — | — | — | 17.8 | — | — | — | — | — | — | — | — | — | |
| Mistral-7BParameters=7.2B, Layers=32, Architecture=Transformer, Pruning Ratio=50% Wanda, Evaluation Tokens=4,0962026.03 | 6.31 | — | — | — | — | — | — | — | — | 11 | — | — | — | — | |
| ADEPTModel Family=Llama, Model Size=13B2026.01 | 6.6 | — | — | — | 20.5 | — | — | — | — | — | — | — | — | — | |
| MIDUS-HMLBackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 7.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MIDUS-HMLBackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 7.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LESABackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 7.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LESABackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 7.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpT-DeUSBackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 7.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpT-DeUSBackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 7.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama ProBackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 7.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama ProBackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 7.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ERMARContext Length=32k, Hardware=NVIDIA L40S GPU2025.03 | 7.88 | — | — | — | — | — | 6 | — | — | — | — | — | — | — | |
| Avg-DeUSBackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 7.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemLongContext Length=32k, Hardware=NVIDIA L40S GPU2025.03 | 7.938 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Avg-DeUSBackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 7.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADEPTModel Family=Llama, Model Size=7B2026.01 | 8.1 | — | — | — | 21.8 | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 8.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 8.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| fp16 baselineCompression=1×, Base Model=TinyLlama-1.1B2026.05 | 8.619 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBQUANTb (bits/coord)=4.00, Compression=4×, k=2, N=256, Base Model=TinyLlama-1.1B2026.05 | 8.715 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBQUANTb (bits/coord)=3.50, Compression=4.6×, k=2, N=128, Base Model=TinyLlama-1.1B2026.05 | 8.881 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TURBOQUANTb (bits/coord)=4.00, Compression=4×, Base Model=TinyLlama-1.1B2026.05 | 9.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INTb (bits/coord)=4.00, Compression=4×, Base Model=TinyLlama-1.1B2026.05 | 9.244 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBQUANTb (bits/coord)=3.00, Compression=5.3×, k=2, N=64, Base Model=TinyLlama-1.1B2026.05 | 9.248 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOLARBackbone=Llama-3.1-8B, Training Stage=SFT2025.12 | 9.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformertotal_params=262M, n_heads=16, MACs=5.4G, Mem (floats)=21M2023.12 | 9.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SwitchAlltotal_params=262M, n_heads=4, MACs=2.4G, Mem (floats)=5.6M2023.12 | 9.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOLARBackbone=Llama-3.1-8B, Training Stage=CPT2025.12 | 9.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBQUANTb (bits/coord)=2.50, Compression=6.4×, k=4, N=1024, Base Model=TinyLlama-1.1B2026.05 | 10.219 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-8BParameters=8B, Layers=36, Architecture=Transformer, Pruning Ratio=50% Wanda, Evaluation Tokens=4,0962026.03 | 10.38 | — | — | — | — | — | — | — | — | 42,922 | — | — | — | — | |
| BaselineNumber of Parameters=7.5B2021.12 | 10.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Megatron-LMParameter Count=8.3B, Zero-shot=true2019.09 | 10.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TURBOQUANTb (bits/coord)=3.00, Compression=5.3×, Base Model=TinyLlama-1.1B2026.05 | 11.413 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SwitchAlltotal_params=47M, n_heads=2, MACs=170M, Mem (floats)=0.8M2023.12 | 12.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformertotal_params=47M, n_heads=10, MACs=453M, Mem (floats)=3.5M2023.12 | 12.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Megatron-LMParameter Count=2.5B, Zero-shot=true2019.09 | 12.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ensemble of AllWeight=12023.11 | 13.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GateLoopParameters=125M2023.11 | 13.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| full-precisionBits (W-E-A)=full-prec., Size (MB)=474.92022.03 | 14.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| full-prec.Size=474.9 MB, Compression Ratio=1.0x2022.03 | 14.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN LMWeight=0.532023.11 | 14.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 14.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=8-8-8, Size=121.4 MB, Compression Ratio=3.9x2022.03 | 14.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADEPTModel Family=GPT2, Model Size=XL2026.01 | 14.7 | — | — | — | 21.4 | — | — | — | — | — | — | — | — | — | |
| LSQBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 15.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=4-4-8, Size=62.4 MB, Compression Ratio=7.6x2022.03 | 15.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=4-4-8, Size (MB)=62.42022.03 | 15.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| adaptive kNN-LMn-gram overlap filtering=false2022.10 | 15.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adaptive kNN-LMλ=lambda_q, b=32, k=10242022.10 | 15.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adaptive kNN-LM (TFIDF)λ=lambda_q, b=32, k=10242022.10 | 15.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IGLUsigma=52026.03 | 15.71 | — | — | — | — | — | — | 2.754 | — | — | — | — | — | — | |
| IGLUsigma=learnable2026.03 | 15.71 | — | — | — | — | — | — | 2.754 | — | — | — | — | — | — | |
| IGLUsigma=102026.03 | 15.72 | — | — | — | — | — | — | 2.755 | — | — | — | — | — | — | |
| ReLU2026.03 | 15.73 | — | — | — | — | — | — | 2.755 | — | — | — | — | — | — | |
| IGLU-APPROXsigma=52026.03 | 15.74 | — | — | — | — | — | — | 2.756 | — | — | — | — | — | — | |
| IGLU-APPROXsigma=102026.03 | 15.74 | — | — | — | — | — | — | 2.756 | — | — | — | — | — | — | |
| IGLU-APPROXsigma=learnable2026.03 | 15.74 | — | — | — | — | — | — | 2.756 | — | — | — | — | — | — | |
| kNN-LM + Continuous Cache# Trainable Params=247M2019.11 | 15.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LMZero-shot=true2019.09 | 15.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LMParameters=247M, Train Speed=13.9k, Inference Mode=Sliding Window, Inference Speed=1452020.12 | 15.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LM + CCacheλ=0.25, b=1, k=10242022.10 | 15.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerLayers=10, Heads=162020.03 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Best Existing2020.02 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerInference Mode=No overlap2020.12 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Routing Trans.2021.05 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Routing Transformer2022.02 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Routing Transformer2021.02 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LM2021.02 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Routing TransformerParams=257M, Context=4096, Tokenizer=Word-level*, Arch. Type=Sparse Attn2026.06 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IGLUsigma=12026.03 | 15.82 | — | — | — | — | — | — | 2.761 | — | — | — | — | — | — | |
| GELU2026.03 | 15.82 | — | — | — | — | — | — | 2.761 | — | — | — | — | — | — | |
| FIBQUANTb (bits/coord)=2.00, Compression=8×, k=4, N=256, Base Model=TinyLlama-1.1B2026.05 | 15.879 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LAQBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 15.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mish2026.03 | 15.9 | — | — | — | — | — | — | 2.766 | — | — | — | — | — | — | |
| GPT2 + GFSA#Params=117M, Fine-tuning=true2023.12 | 15.919 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SiLU2026.03 | 15.92 | — | — | — | — | — | — | 2.767 | — | — | — | — | — | — | |
| GPT2#Params=117M, Fine-tuning=true2023.12 | 15.939 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IGLU-APPROXsigma=12026.03 | 15.99 | — | — | — | — | — | — | 2.772 | — | — | — | — | — | — | |
| kNN-LM# Trainable Params=247M, base_model=Baevski & Auli (2019)2019.11 | 16.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LMn-gram overlap filtering=false2022.10 | 16.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| kNN-LMλ=0.25, b=1, k=10242022.10 | 16.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hardswish2026.03 | 16.15 | — | — | — | — | — | — | 2.781 | — | — | — | — | — | — | |
| ADEPTModel Family=GPT2, Model Size=Large2026.01 | 16.3 | — | — | — | 29.6 | — | — | — | — | — | — | — | — | — | |
| MambaNumber of Parameters=130M, Tokenizer=same2024.10 | 16.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MambaNumber of Parameters=130M2024.10 | 16.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LightPAFFSize=268.0 MB, Compression Ratio=1.8x2022.03 | 16.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PHA-180MParams=180M, Context=1024, Tokenizer=GPT-2 BPE, Arch. Type=Hybrid2026.06 | 16.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PHA-125MParams=125M, Context=1024, Tokenizer=GPT-2 BPE, Arch. Type=Hybrid (GSS+Attn)2026.06 | 16.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IGLU-APPROXsigma=0.52026.03 | 16.6 | — | — | — | — | — | — | 2.809 | — | — | — | — | — | — | |
| IGLUsigma=0.12026.03 | 16.69 | — | — | — | — | — | — | 2.989 | — | — | — | — | — | — | |
| IGLUsigma=0.52026.03 | 16.69 | — | — | — | — | — | — | 2.814 | — | — | — | — | — | — | |
| HedgehogParams=125M, Context=1024, Tokenizer=GPT-2 BPE, Arch. Type=Linear Attn2026.06 | 16.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PACTBits (W-E-A)=8-8-8, Size (MB)=121.42022.03 | 16.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| H3-355MParams=355M, Context=2048, Tokenizer=GPT-2 BPE, Arch. Type=Hybrid SSM2026.06 | 16.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LAQBits (W-E-A)=4-4-8, Size (MB)=62.42022.03 | 16.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuantGPTBits (W-E-A)=2-2-8, Size (MB)=33.02022.03 | 16.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Segatron-XL large + HCP#Param.=257M, Memory length (M)=384, Training steps=350k2022.03 | 17 | — | — | — | — | — | — | — | — | — | — | — | — | — |