Language Modeling on Wikitext (PPL)
7.21Wikitext PPLBaseline
Evaluation results
| Method | Links | |
|---|---|---|
| BaselineBackbone=Llama 3.1 8B Instruct, Compression Ratio=1.00x, Post-compression fine-tuning=false2026.06 | 7.21 | |
| ASVD+Backbone=Llama 3.1 8B Instruct, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 8.26 | |
| SigmaScale (KD)Backbone=Llama 3.1 8B Instruct, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 8.7 | |
| SigmaScaleBackbone=Llama 3.1 8B Instruct, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 8.95 | |
| BaselineBackbone=Qwen3-8B, Compression Ratio=1.00x, Post-compression fine-tuning=false2026.06 | 9.72 | |
| ASVD+Backbone=Qwen3-8B, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 10.11 | |
| FP16#Bits (W-A-KV)=16-16-16, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 10.48 | |
| FP16#Bits (W-A-KV)=16-16-16, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 10.48 | |
| FP16#Bits (W-A-KV)=16-16-16, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 10.48 | |
| FPTQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 10.56 | |
| FlatQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 10.68 | |
| FlatQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 10.68 | |
| FPTQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 10.78 | |
| SigmaScale (KD)Backbone=Qwen3-8B, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 10.84 | |
| SpinQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 10.88 | |
| SigmaScaleBackbone=Qwen3-8B, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 10.89 | |
| FPTQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 10.99 | |
| FlatQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 11.38 | |
| FlatQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 11.49 | |
| SVD-LLMBackbone=Qwen3-8B, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 11.51 | |
| SpinQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 11.71 | |
| FPTQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 11.71 | |
| SpinQuant#Bits (W-A-KV)=4-8-8, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 11.73 | |
| FlatQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 12.3 | |
| ASVD+Backbone=Qwen3-8B, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 12.34 | |
| SpinQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(i) Linears+KV, Backbone=Llama-3.2 3B instruct2025.06 | 12.71 | |
| Hybrid Gated DeltaNet + M2RNN-5architecture=7B (1B active) MoE, setting=0-shot2026.03 | 12.85 | |
| Hybrid M2RNNarchitecture=7B (1B active) MoE, setting=0-shot2026.03 | 13 | |
| Hybrid Mamba-2 + M2RNN-1architecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.01 | |
| Hybrid Mamba-2 + M2RNN-5architecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.02 | |
| Hybrid Gated DeltaNet + M2RNN-1architecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.07 | |
| Hybrid Mamba-2architecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.1 | |
| SpinQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 13.16 | |
| SVD-LLMBackbone=Llama 3.1 8B Instruct, Compression Ratio=0.90x, Post-compression fine-tuning=true2026.06 | 13.31 | |
| Hybrid Gated DeltaNetarchitecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.51 | |
| SVD-LLMBackbone=Qwen3-8B, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 13.64 | |
| ASVD+Backbone=Llama 3.1 8B Instruct, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 13.67 | |
| Mamba-2architecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.73 | |
| M2RNNarchitecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.8 | |
| Gated DeltaNetarchitecture=7B (1B active) MoE, setting=0-shot2026.03 | 13.89 | |
| FPTQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(ii) Linears+KV+BMM inputs, Backbone=Llama-3.2 3B instruct2025.06 | 13.99 | |
| SigmaScale (KD)Backbone=Qwen3-8B, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 14.43 | |
| SigmaScaleBackbone=Qwen3-8B, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 14.68 | |
| GRUarchitecture=7B (1B active) MoE, setting=0-shot2026.03 | 14.8 | |
| Transformer++architecture=7B (1B active) MoE, setting=0-shot2026.03 | 14.94 | |
| SoftmaxQuantization=8-bit2025.04 | 15.1 | |
| SoftmaxModel Scale=1.8B2025.04 | 15.1 | |
| SoftmaxQuantization=2-bit2025.04 | 15.1 | |
| SoftmaxQuantization=3-bit2025.04 | 15.1 | |
| FullAttnInference Mode=Full Attention Inference2025.11 | 15.13 | |
| SSAInference Mode=Full Attention Inference2025.11 | 15.28 | |
| NSAInference Mode=Sparse Attention Inference, Receptive Field=10242025.11 | 15.48 | |
| SSAInference Mode=Sparse Attention Inference, Receptive Field=10242025.11 | 15.51 | |
| FullAttnInference Mode=Sparse Attention Inference, Receptive Field=10242025.11 | 15.67 | |
| SoftmaxQuantization=4-bit2025.04 | 15.84 | |
| SSAInference Mode=Sparse Attention Inference, Receptive Field=2562025.11 | 15.96 | |
| NSAInference Mode=Sparse Attention Inference, Receptive Field=2562025.11 | 16.02 | |
| MoBAInference Mode=Sparse Attention Inference, Receptive Field=10242025.11 | 16.02 | |
| MoBAInference Mode=Sparse Attention Inference, Receptive Field=2562025.11 | 16.54 | |
| MoBAInference Mode=Full Attention Inference2025.11 | 16.68 | |
| FullAttnInference Mode=Sparse Attention Inference, Receptive Field=2562025.11 | 17.05 | |
| FPTQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 17.17 | |
| RNNarchitecture=7B (1B active) MoE, setting=0-shot2026.03 | 17.65 | |
| SoftpickQuantization=8-bit2025.04 | 17.86 | |
| SoftpickQuantization=2-bit2025.04 | 17.86 | |
| SoftpickQuantization=3-bit2025.04 | 17.86 | |
| SoftpickModel Scale=1.8B2025.04 | 17.87 | |
| SigmaScale (KD)Backbone=Llama 3.1 8B Instruct, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 17.9 | |
| GPTParameters=1.3B, Training tokens=100B2025.11 | 17.93 | |
| SVD-LLMBackbone=Llama 3.1 8B Instruct, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 18.15 | |
| NextLat (d=1)Horizon (d)=1, Parameters=1.3B, Training tokens=100B2025.11 | 18.39 | |
| NextLat (d=2)Horizon (d)=2, Parameters=1.3B, Training tokens=100B2025.11 | 18.44 | |
| SoftpickQuantization=4-bit2025.04 | 18.46 | |
| SigmaScaleBackbone=Llama 3.1 8B Instruct, Compression Ratio=0.75x, Post-compression fine-tuning=true2026.06 | 18.48 | |
| FlatQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 18.6 | |
| MTP (d=2)Horizon (d)=2, Parameters=1.3B, Training tokens=100B2025.11 | 18.61 | |
| MTP (d=1)Horizon (d)=1, Parameters=1.3B, Training tokens=100B2025.11 | 18.82 | |
| SoftmaxQuantization=3-bit2025.04 | 19.24 | |
| JTP (d=1)Horizon (d)=1, Parameters=1.3B, Training tokens=100B2025.11 | 19.28 | |
| JTP (d=2)Horizon (d)=2, Parameters=1.3B, Training tokens=100B2025.11 | 19.6 | |
| SpinQuant#Bits (W-A-KV)=4-4-4, Activation Setting=(iii) All activations except residual, Backbone=Llama-3.2 3B instruct2025.06 | 20.13 | |
| Hybrid Gated DeltaNet + M2RNN-3Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.26 | |
| Hybrid Mamba-2 + M2RNN-3Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.39 | |
| Hybrid Gated DeltaNet + M2RNN-1Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.39 | |
| Hybrid Mamba-2 + M2RNN-1Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.48 | |
| Hybrid M2RNNModel size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.53 | |
| Hybrid Mamba-2Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.59 | |
| SoftpickQuantization=3-bit2025.04 | 21.77 | |
| SVD-LLMBackbone=Qwen3-8B, Compression Ratio=0.50x, Post-compression fine-tuning=true2026.06 | 21.84 | |
| Hybrid Gated DeltaNetModel size=410M parameter dense model, Setting=0-shot setting2026.03 | 21.89 | |
| Gated DeltaNetModel size=410M parameter dense model, Setting=0-shot setting2026.03 | 22.7 | |
| M2RNNModel size=410M parameter dense model, Setting=0-shot setting2026.03 | 22.92 | |
| Mamba-2Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 22.93 | |
| Transformer++Model size=410M parameter dense model, Setting=0-shot setting2026.03 | 23.32 | |
| SoftmaxQuantization=8-bit2025.04 | 23.84 | |
| SoftmaxModel Scale=340M2025.04 | 23.85 | |
| SoftmaxQuantization=8-bit2025.04 | 23.95 | |
| ASVD+Backbone=Qwen3-8B, Compression Ratio=0.50x, Post-compression fine-tuning=true2026.06 | 24.3 | |
| SoftpickQuantization=8-bit2025.04 | 24.31 | |
| SoftpickModel Scale=340M2025.04 | 24.32 |