Language Modeling on WikiText-103 (Delta PPL)
0Delta PPLGAIN
Evaluation results
| Method | Links | |
|---|---|---|
| GAINLR=3×10−4, Params=46K2026.04 | 0 | |
| LoRALR=10−4, Params=92K2026.04 | 0 | |
| LO-BCQ (g128)Backbone=Llama2-70B, Bitwidth=4.56, #codebooks=162025.02 | 0.05 | |
| LO-BCQ (g128)Backbone=Llama2-70B, Bitwidth=4.44, #codebooks=82025.02 | 0.06 | |
| AQLMBackbone=Llama2-70B, Bitwidth=4.14, #codebooks=2 * 2^162025.02 | 0.07 | |
| LO-BCQ (g128)Backbone=Llama2-70B, Bitwidth=4.31, #codebooks=42025.02 | 0.07 | |
| LO-BCQ (g128)Backbone=Llama2-7B, Bitwidth=4.56, #codebooks=162025.02 | 0.08 | |
| AWQ (g128)Backbone=Llama2-70B, Bitwidth=4.132025.02 | 0.09 | |
| AQLMBackbone=Llama2-7B, Bitwidth=4.14, #codebooks=2 * 2^162025.02 | 0.09 | |
| LO-BCQ (g128)Backbone=Llama2-70B, Bitwidth=4.19, #codebooks=22025.02 | 0.09 | |
| LO-BCQ (g128)Backbone=Llama2-7B, Bitwidth=4.44, #codebooks=82025.02 | 0.09 | |
| QuiP#Backbone=Llama2-70B, Bitwidth=4.02, #codebooks=2 * 2^162025.02 | 0.1 | |
| GAINLR=10−3, Params=46K2026.04 | 0.1 | |
| GAIN-FFNLR=10−3, Params=230K2026.04 | 0.1 | |
| LoRALR=3×10−4, Params=92K2026.04 | 0.1 | |
| LO-BCQ (g128)Backbone=Llama2-7B, Bitwidth=4.31, #codebooks=42025.02 | 0.12 | |
| AWQ (g128)Backbone=Llama2-7B, Bitwidth=4.132025.02 | 0.13 | |
| LO-BCQ (g128)Backbone=Llama2-7B, Bitwidth=4.19, #codebooks=22025.02 | 0.14 | |
| QuiP#Backbone=Llama2-7B, Bitwidth=4.02, #codebooks=2 * 2^162025.02 | 0.17 | |
| GPTQ (g128)Backbone=Llama2-70B, Bitwidth=42025.02 | 0.23 | |
| GAINLR=3×10−3, Params=46K2026.04 | 0.3 | |
| GPTQ (g128)Backbone=Llama2-7B, Bitwidth=42025.02 | 0.37 | |
| GAIN-FFNLR=3×10−3, Params=230K2026.04 | 1 | |
| LoRALR=10−3, Params=92K2026.04 | 1.1 | |
| LoRALR=10−2, Params=92K2026.04 | 10.4 |