Language Modeling on WikiText-2 (test) (Context Length 4096)
5.11PPL (WikiText-2)Baseline (Ours)
Evaluation results
| Method | Links | |
|---|---|---|
| Baseline (Ours)Finetuned=No, LM Eval=acc, BPW=162026.03 | 5.11 | |
| LLVQ [shape-gain, 2 bit gain]Finetuned=Yes, LM Eval=acc, BPW=22026.03 | 5.48 | |
| LLVQ [spherical shaping]Finetuned=Yes, LM Eval=acc, BPW=22026.03 | 5.6 | |
| PV-tuningFinetuned=Yes, LM Eval=acc, BPW=2, Details=Table 8 of PV-tuning paper2026.03 | 5.84 | |
| QTIPFinetuned=Yes, LM Eval=not reported, BPW=2, Details=Tables 5 & 6 of QTIP paper2026.03 | 5.86 | |
| Quip#Finetuned=Yes, LM Eval=not reported, BPW=2, Details=Tables 5 & 6 of QTIP paper2026.03 | 6.19 | |
| Gemma2-9B (Baseline)MP=false, bits=16, Backbone=Gemma2-9B2026.02 | 6.36 | |
| ScaleBITS + RTNMP=true, bits=3.1, Backbone=Gemma2-9B2026.02 | 6.74 | |
| LLVQ [shape-gain, 2 bit gain]Finetuned=No, LM Eval=acc, BPW=22026.03 | 6.83 | |
| AQLMFinetuned=Yes, LM Eval=not reported, BPW=2.07, Details=Tables 5 & 6 of QTIP paper2026.03 | 6.93 | |
| LLVQ [spherical shaping]Finetuned=No, LM Eval=acc, BPW=22026.03 | 7.61 | |
| RTN-g128MP=false, bits=3.1, Backbone=Gemma2-9B2026.02 | 8.11 | |
| Quip#Finetuned=No, LM Eval=acc/acc_norm, BPW=2, Details=Tables 4 & 10 of Quip# paper2026.03 | 8.22 | |
| ScaleBITS + RTNMP=true, bits=2.1, Backbone=Gemma2-9B2026.02 | 8.89 | |
| RTN-g128MP=false, bits=2.1, Backbone=Gemma2-9B2026.02 | 30,000 |