Language Modeling on WikiText-103 (Perplexity)
15.33PerplexityMCWC
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MCWCCompression Ratio=2×, Bits per model parameter=16.6, Storage (MB)=2910, Decode time (s)=4.22026.05 | 15.33 | — | |
| MCWCCompression Ratio=4×, Bits per model parameter=8.3, Storage (MB)=1460, Decode time (s)=3.12026.05 | 15.46 | — | |
| QAT INT8Compression Ratio=4×, Bits per model parameter=8.4, Storage (MB)=1480, Decode time (s)=3.02026.05 | 15.62 | — | |
| SmoothQuant INT8Compression Ratio=4×, Bits per model parameter=8.4, Storage (MB)=1485, Decode time (s)=3.02026.05 | 15.7 | — | |
| Per-channel PTQ INT8Compression Ratio=4×, Bits per model parameter=8.4, Storage (MB)=1480, Decode time (s)=2.92026.05 | 15.78 | — | |
| Uniform PTQ INT8Compression Ratio=4×, Bits per model parameter=8.6, Storage (MB)=1515, Decode time (s)=2.82026.05 | 15.92 | — | |
| MCWCCompression Ratio=8×, Bits per model parameter=4.2, Storage (MB)=740, Decode time (s)=2.52026.05 | 15.95 | — | |
| AWQ W4Compression Ratio=8×, Bits per model parameter=4.2, Storage (MB)=745, Decode time (s)=2.62026.05 | 16.25 | — | |
| QAT INT4Compression Ratio=8×, Bits per model parameter=4.3, Storage (MB)=760, Decode time (s)=2.42026.05 | 16.35 | — | |
| MCWCCompression Ratio=12×, Bits per model parameter=2.8, Storage (MB)=495, Decode time (s)=2.32026.05 | 16.4 | — | |
| GPTQ W4Compression Ratio=8×, Bits per model parameter=4.3, Storage (MB)=755, Decode time (s)=2.62026.05 | 16.4 | — | |
| Uniform PTQ INT4Compression Ratio=8×, Bits per model parameter=4.3, Storage (MB)=760, Decode time (s)=2.32026.05 | 16.85 | — | |
| Learned SQ + entropyCompression Ratio=12×, Bits per model parameter=2.9, Storage (MB)=520, Decode time (s)=2.72026.05 | 17.3 | — | |
| Prune+Quant (50%)Compression Ratio=12×, Bits per model parameter=2.7, Storage (MB)=480, Decode time (s)=2.12026.05 | 18.1 | — | |
| Qwen3-0.6B (Pretrained)2026.05 | 21.5 | — | |
| DLR-NetDistillation stage=+Logits distillation (Phase 2)2026.05 | 23.4 | — | |
| DLR-NetDistillation stage=Module-wise distillation (Phase 1)2026.05 | 182 | — | |
| Hallamonic 1BSequence Length=1,0242026.05 | — | 0.4321 | |
| Hallamonic 1BSequence Length=4,0962026.05 | — | 0.4525 | |
| Hallamonic 1BSequence Length=8,1922026.05 | — | 0.4774 | |
| TinyLlama 1.1BSequence Length=1,0242026.05 | — | 2.8436 | |
| TinyLlama 1.1BSequence Length=4,0962026.05 | — | 7.3188 | |
| TinyLlama 1.1BSequence Length=8,1922026.05 | — | 10.5564 |