Language Modeling on Wikitext-103 (PPL and Delta PPL)
3.14PPLBF16 (Pretrained)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| BF16 (Pretrained)Bitwidth (W4A4)=16, Model Name=Llama2, Model Size=70B2025.02 | 3.14 | — | — | — | |
| LO-BCQBitwidth (W4A4)=4.75, Group Size=g32, Nc=16, Model Name=Llama2, Model Size=70B2025.02 | 3.2 | 0.06 | — | — | |
| LO-BCQBitwidth (W4A4)=4.5, Group Size=g64, Nc=8, Model Name=Llama2, Model Size=70B2025.02 | 3.23 | 0.09 | — | — | |
| LO-BCQBitwidth (W4A4)=4.25, Group Size=g64, Nc=2, Model Name=Llama2, Model Size=70B2025.02 | 3.35 | 0.21 | — | — | |
| BF16 (Pretrained)Bitwidth (W4A4)=16, Model Name=Nemotron4, Model Size=340B2025.02 | 3.48 | — | — | — | |
| LO-BCQBitwidth (W4A4)=4.75, Group Size=g32, Nc=16, Model Name=Nemotron4, Model Size=340B2025.02 | 3.56 | 0.08 | — | — | |
| MX4Bitwidth (W4A4)=4.5, Group Size=g16, Model Name=Llama2, Model Size=70B2025.02 | 3.58 | 0.44 | — | — | |
| LO-BCQBitwidth (W4A4)=4.5, Group Size=g64, Nc=8, Model Name=Nemotron4, Model Size=340B2025.02 | 3.6 | 0.12 | — | — | |
| LO-BCQBitwidth (W4A4)=4.25, Group Size=g64, Nc=2, Model Name=Nemotron4, Model Size=340B2025.02 | 3.67 | 0.19 | — | — | |
| MXFP4Bitwidth (W4A4)=4.25, Group Size=g32, Model Name=Llama2, Model Size=70B2025.02 | 3.69 | 0.55 | — | — | |
| MX4Bitwidth (W4A4)=4.5, Group Size=g16, Model Name=Nemotron4, Model Size=340B2025.02 | 4.01 | 0.53 | — | — | |
| MXFP4Bitwidth (W4A4)=4.25, Group Size=g32, Model Name=Nemotron4, Model Size=340B2025.02 | 4.1 | 0.62 | — | — | |
| VSQBitwidth (W4A4)=4.5, Group Size=g16, Model Name=Nemotron4, Model Size=340B2025.02 | 4.19 | 0.71 | — | — | |
| VSQBitwidth (W4A4)=4.5, Group Size=g16, Model Name=Llama2, Model Size=70B2025.02 | 4.96 | 1.82 | — | — | |
| BF16 (Pretrained)Bitwidth (W4A4)=16, Model Name=Llama2, Model Size=7B2025.02 | 5.06 | — | — | — | |
| LO-BCQBitwidth (W4A4)=4.75, Group Size=g32, Nc=16, Model Name=Llama2, Model Size=7B2025.02 | 5.15 | 0.09 | — | — | |
| LO-BCQBitwidth (W4A4)=4.5, Group Size=g64, Nc=8, Model Name=Llama2, Model Size=7B2025.02 | 5.19 | 0.13 | — | — | |
| LO-BCQBitwidth (W4A4)=4.25, Group Size=g64, Nc=2, Model Name=Llama2, Model Size=7B2025.02 | 5.31 | 0.25 | — | — | |
| MX4Bitwidth (W4A4)=4.5, Group Size=g16, Model Name=Llama2, Model Size=7B2025.02 | 5.73 | 0.67 | — | — | |
| MXFP4Bitwidth (W4A4)=4.25, Group Size=g32, Model Name=Llama2, Model Size=7B2025.02 | 5.76 | 0.7 | — | — | |
| BF16 (Pretrained)Bitwidth (W4A4)=16, Model Name=Nemotron4, Model Size=15B2025.02 | 5.87 | — | — | — | |
| LO-BCQBitwidth (W4A4)=4.75, Group Size=g32, Nc=16, Model Name=Nemotron4, Model Size=15B2025.02 | 6.03 | 0.16 | — | — | |
| LO-BCQBitwidth (W4A4)=4.5, Group Size=g64, Nc=8, Model Name=Nemotron4, Model Size=15B2025.02 | 6.13 | 0.26 | — | — | |
| LO-BCQBitwidth (W4A4)=4.25, Group Size=g64, Nc=2, Model Name=Nemotron4, Model Size=15B2025.02 | 6.3 | 0.43 | — | — | |
| BF16 (Pretrained)Bitwidth (W4A4)=16, Model Name=GPT3, Model Size=22B2025.02 | 6.54 | — | — | — | |
| LO-BCQBitwidth (W4A4)=4.75, Group Size=g32, Nc=16, Model Name=GPT3, Model Size=22B2025.02 | 6.59 | 0.05 | — | — | |
| LO-BCQBitwidth (W4A4)=4.5, Group Size=g64, Nc=8, Model Name=GPT3, Model Size=22B2025.02 | 6.62 | 0.08 | — | — | |
| LO-BCQBitwidth (W4A4)=4.25, Group Size=g64, Nc=2, Model Name=GPT3, Model Size=22B2025.02 | 6.74 | 0.2 | — | — | |
| VSQBitwidth (W4A4)=4.5, Group Size=g16, Model Name=GPT3, Model Size=22B2025.02 | 7.12 | 0.58 | — | — | |
| BF16 (Pretrained)Bitwidth (W4A4)=16, Model Name=GPT3, Model Size=8B2025.02 | 7.38 | — | — | — | |
| LO-BCQBitwidth (W4A4)=4.75, Group Size=g32, Nc=16, Model Name=GPT3, Model Size=8B2025.02 | 7.45 | 0.07 | — | — | |
| LO-BCQBitwidth (W4A4)=4.5, Group Size=g64, Nc=8, Model Name=GPT3, Model Size=8B2025.02 | 7.48 | 0.1 | — | — | |
| VSQBitwidth (W4A4)=4.5, Group Size=g16, Model Name=Nemotron4, Model Size=15B2025.02 | 7.58 | 1.71 | — | — | |
| LO-BCQBitwidth (W4A4)=4.25, Group Size=g64, Nc=2, Model Name=GPT3, Model Size=8B2025.02 | 7.61 | 0.23 | — | — | |
| MX4Bitwidth (W4A4)=4.5, Group Size=g16, Model Name=GPT3, Model Size=22B2025.02 | 7.69 | 1.15 | — | — | |
| MX4Bitwidth (W4A4)=4.5, Group Size=g16, Model Name=GPT3, Model Size=8B2025.02 | 8.15 | 0.77 | — | — | |
| VSQBitwidth (W4A4)=4.5, Group Size=g16, Model Name=GPT3, Model Size=8B2025.02 | 8.17 | 0.79 | — | — | |
| MXFP4Bitwidth (W4A4)=4.25, Group Size=g32, Model Name=Nemotron4, Model Size=15B2025.02 | 8.24 | 2.37 | — | — | |
| MX4Bitwidth (W4A4)=4.5, Group Size=g16, Model Name=Nemotron4, Model Size=15B2025.02 | 8.88 | 3.01 | — | — | |
| MXFP4Bitwidth (W4A4)=4.25, Group Size=g32, Model Name=GPT3, Model Size=8B2025.02 | 9.12 | 1.74 | — | — | |
| MXFP4Bitwidth (W4A4)=4.25, Group Size=g32, Model Name=GPT3, Model Size=22B2025.02 | 10.18 | 3.64 | — | — | |
| VSQBitwidth (W4A4)=4.5, Group Size=g16, Model Name=Llama2, Model Size=7B2025.02 | 835 | 829 | — | — | |
| LO-BCQBitwidth (W4A4)=4.19, Group size=g128, Nc (Number of codebooks)=22025.02 | — | — | 0.14 | 0.09 | |
| LO-BCQBitwidth (W4A4)=4.31, Group size=g128, Nc (Number of codebooks)=42025.02 | — | — | 0.12 | 0.07 | |
| LO-BCQBitwidth (W4A4)=4.44, Group size=g128, Nc (Number of codebooks)=82025.02 | — | — | 0.09 | 0.06 | |
| LO-BCQBitwidth (W4A4)=4.56, Group size=g128, Nc (Number of codebooks)=162025.02 | — | — | 0.08 | 0.05 |