Language Modeling Statistics on WikiText-2
93.039Average KurtosisVanilla
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| VanillaModel=Qwen3-0.6B, Precision=FP162026.05 | 93.039 | 74.778 | 10.568 | |
| VanillaModel=Qwen3-0.6B, Precision=W8A82026.05 | 93.039 | 74.778 | 36.642 | |
| VanillaModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 93.039 | 74.778 | 44.778 | |
| VanillaModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 93.039 | 74.778 | 50.821 | |
| VanillaModel=Llama-3.2-1B, Precision=FP162026.05 | 91.706 | 84.433 | 8.981 | |
| VanillaModel=Llama-3.2-1B, Precision=W8A82026.05 | 91.706 | 84.433 | 30.189 | |
| VanillaModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 91.706 | 84.433 | 39.378 | |
| VanillaModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 91.706 | 84.433 | 74.967 | |
| Gated AttentionModel=Llama-3.2-1B, Precision=FP162026.05 | 4.394 | 14.889 | 8.259 | |
| Gated AttentionModel=Llama-3.2-1B, Precision=W8A82026.05 | 4.394 | 14.889 | 9.214 | |
| Gated AttentionModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 4.394 | 14.889 | 13.103 | |
| Gated AttentionModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 4.394 | 14.889 | 24.27 | |
| Clipped SoftmaxModel=Llama-3.2-1B, Precision=FP162026.05 | 3.297 | 14.243 | 9.049 | |
| Clipped SoftmaxModel=Llama-3.2-1B, Precision=W8A82026.05 | 3.297 | 14.243 | 10.264 | |
| Clipped SoftmaxModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 3.297 | 14.243 | 14.368 | |
| Clipped SoftmaxModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 3.297 | 14.243 | 26.808 | |
| OutEffHopModel=Llama-3.2-1B, Precision=FP162026.05 | 3.076 | 13.776 | 9.07 | |
| OutEffHopModel=Llama-3.2-1B, Precision=W8A82026.05 | 3.076 | 13.776 | 10.419 | |
| OutEffHopModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 3.076 | 13.776 | 13.555 | |
| OutEffHopModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 3.076 | 13.776 | 26.183 | |
| OutEffHopModel=Qwen3-0.6B, Precision=FP162026.05 | 3.035 | 14.164 | 9.991 | |
| OutEffHopModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.035 | 14.164 | 10.599 | |
| OutEffHopModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.035 | 14.164 | 13.357 | |
| OutEffHopModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.035 | 14.164 | 32.362 | |
| Gated AttentionModel=Qwen3-0.6B, Precision=FP162026.05 | 3.033 | 14.36 | 10.556 | |
| Gated AttentionModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.033 | 14.36 | 19.261 | |
| Gated AttentionModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.033 | 14.36 | 23.427 | |
| Gated AttentionModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.033 | 14.36 | 28.775 | |
| AoSModel=Llama-3.2-1B, Precision=FP162026.05 | 3.031 | 12.538 | 9 | |
| AoSModel=Llama-3.2-1B, Precision=W8A82026.05 | 3.031 | 12.538 | 9.364 | |
| AoSModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 3.031 | 12.538 | 11.284 | |
| AoSModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 3.031 | 12.538 | 22.633 | |
| AoSModel=Qwen3-0.6B, Precision=FP162026.05 | 3.031 | 13.898 | 9.345 | |
| AoSModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.031 | 13.898 | 9.615 | |
| AoSModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.031 | 13.898 | 10.484 | |
| AoSModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.031 | 13.898 | 17.516 | |
| Gated OutEffHopModel=Qwen3-0.6B, Precision=FP162026.05 | 3.029 | 13.81 | 9.999 | |
| Gated OutEffHopModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.029 | 13.81 | 10.572 | |
| Gated OutEffHopModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.029 | 13.81 | 11.521 | |
| Gated OutEffHopModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.029 | 13.81 | 21.258 | |
| Clipped SoftmaxModel=Qwen3-0.6B, Precision=FP162026.05 | 3.027 | 14.164 | 10.366 | |
| Clipped SoftmaxModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.027 | 14.164 | 21.382 | |
| Clipped SoftmaxModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.027 | 14.164 | 28.736 | |
| Clipped SoftmaxModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.027 | 14.164 | 39.891 | |
| Gated AoSModel=Qwen3-0.6B, Precision=FP162026.05 | 3.025 | 13.55 | 9.353 | |
| Gated AoSModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.025 | 13.55 | 9.591 | |
| Gated AoSModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.025 | 13.55 | 10.455 | |
| Gated AoSModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.025 | 13.55 | 15.432 | |
| Clipped OutEffHopModel=Qwen3-0.6B, Precision=FP162026.05 | 3.023 | 12.685 | 9.869 | |
| Clipped OutEffHopModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.023 | 12.685 | 10.437 | |
| Clipped OutEffHopModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.023 | 12.685 | 11.385 | |
| Clipped OutEffHopModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.023 | 12.685 | 30.742 | |
| Clipped AoSModel=Qwen3-0.6B, Precision=FP162026.05 | 3.019 | 12.45 | 9.231 | |
| Clipped AoSModel=Qwen3-0.6B, Precision=W8A82026.05 | 3.019 | 12.45 | 9.469 | |
| Clipped AoSModel=Qwen3-0.6B, Precision=OmniQuant W6A62026.05 | 3.019 | 12.45 | 10.326 | |
| Clipped AoSModel=Qwen3-0.6B, Precision=OmniQuant W4A42026.05 | 3.019 | 12.45 | 16.949 | |
| Clipped OutEffHopModel=Llama-3.2-1B, Precision=FP162026.05 | 2.399 | 10.848 | 9.08 | |
| Clipped OutEffHopModel=Llama-3.2-1B, Precision=W8A82026.05 | 2.399 | 10.848 | 10.188 | |
| Clipped OutEffHopModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 2.399 | 10.848 | 13.277 | |
| Clipped OutEffHopModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 2.399 | 10.848 | 25.624 | |
| Clipped AoSModel=Llama-3.2-1B, Precision=FP162026.05 | 2.364 | 9.874 | 9.01 | |
| Clipped AoSModel=Llama-3.2-1B, Precision=W8A82026.05 | 2.364 | 9.874 | 9.336 | |
| Clipped AoSModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 2.364 | 9.874 | 11.25 | |
| Clipped AoSModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 2.364 | 9.874 | 22.565 | |
| Gated OutEffHopModel=Llama-3.2-1B, Precision=FP162026.05 | 2.139 | 8.325 | 8.751 | |
| Gated OutEffHopModel=Llama-3.2-1B, Precision=W8A82026.05 | 2.139 | 8.325 | 9.198 | |
| Gated OutEffHopModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 2.139 | 8.325 | 12.083 | |
| Gated OutEffHopModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 2.139 | 8.325 | 23.231 | |
| Gated AoSModel=Llama-3.2-1B, Precision=FP162026.05 | 2.108 | 8.026 | 8.683 | |
| Gated AoSModel=Llama-3.2-1B, Precision=W8A82026.05 | 2.108 | 8.026 | 8.875 | |
| Gated AoSModel=Llama-3.2-1B, Precision=OmniQuant W6A62026.05 | 2.108 | 8.026 | 10.695 | |
| Gated AoSModel=Llama-3.2-1B, Precision=OmniQuant W4A42026.05 | 2.108 | 8.026 | 21.451 |