Language Modeling on Wikitext-2 (test) (Llama 2 & 3 Specific Metrics)
5.47PPL (Llama 2, Config 7)FP16
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| FP16Bits=16, context length=20482024.07 | 5.47 | 4.88 | 3.32 | 6.14 | 2.85 | |
| EfficientQATBits=3, Group=128, context length=20482024.07 | 5.81 | 5.12 | 3.61 | 7.09 | 4.21 | |
| BitDistillerBits=3, Group=128, context length=20482024.07 | 5.97 | — | — | — | — | |
| OmniQBits=3, Group=128, context length=20482024.07 | 6.03 | 5.28 | 3.78 | 8.27 | 4.99 | |
| AWQBits=3, Group=128, context length=20482024.07 | 6.24 | 5.32 | 3.74 | 8.16 | 4.69 | |
| GPTQBits=3, Group=128, context length=20482024.07 | 6.29 | 5.42 | 3.85 | 9.58 | 5.25 | |
| QuIP#Bits=2, context length=20482024.07 | 6.66 | 5.74 | 4.16 | — | — | |
| EfficientQATBits=2, Group=64, context length=20482024.07 | 6.86 | 5.96 | 4.52 | 9.41 | 6.07 | |
| EfficientQATBits=2, Group=128, context length=20482024.07 | 7.19 | 6.08 | 4.61 | 9.8 | 6.38 | |
| DB-LLMBits=2, Group=64, context length=20482024.07 | 7.23 | 6.19 | 4.64 | 13.6 | — | |
| AQLMBits=2, Group=2x8, context length=20482024.07 | 7.24 | 6.06 | 4.49 | — | — | |
| ApiQBits=2, Group=64, context length=20482024.07 | 7.59 | 6.44 | — | — | — | |
| CBQBits=2, Group=64, context length=20482024.07 | 8.01 | — | — | — | — | |
| BitDistillerBits=2, Group=128, context length=20482024.07 | 8.08 | — | — | — | — | |
| ApiQBits=2, Group=128, context length=20482024.07 | 8.25 | 6.71 | — | — | — | |
| OmniQBits=2, Group=128, context length=20482024.07 | 11.06 | 8.26 | 6.55 | 18.5 | 16.79 |