Language Modeling on WikiText-103 (PPL, Bits, and Delta)
5.29Perplexity (PPL)fp16
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| fp16Model=Mistral-7B, Arch=dense MHA2026.05 | 5.29 | — | — | |
| Best HQMQModel=Mistral-7B, Arch=dense MHA2026.05 | 5.32 | 5.07 | 0.5 | |
| fp16Model=Llama-3-8B, Arch=dense GQA2026.05 | 6.28 | — | — | |
| Best HQMQModel=Llama-3-8B, Arch=dense GQA2026.05 | 6.32 | 5 | 0.6 | |
| fp16Model=Qwen2.5-7B, Arch=outlier GQA2026.05 | 7.59 | — | — | |
| Best HQMQModel=Qwen2.5-7B, Arch=outlier GQA2026.05 | 8.83 | 5.15 | 16 | |
| fp16Model=Qwen3-8B, Arch=outlier GQA2026.05 | 9.6 | — | — | |
| Best HQMQModel=Qwen3-8B, Arch=outlier GQA2026.05 | 9.62 | 4.99 | 0.2 | |
| fp16Model=gpt-oss-20b†, Arch=sparse MoE2026.05 | 446.8 | — | — | |
| Best HQMQModel=gpt-oss-20b†, Arch=sparse MoE2026.05 | 460.4 | 5.25 | 3 |