Language Modeling on FineWeb-Edu 300M×5B (val)
23.6PPLFP16 reference
Evaluation Results
| Method | Links | |
|---|---|---|
| FP16 referencek=0, Quantization=FP162026.04 | 23.6 | |
| FP16 referencek=64, Quantization=FP16, Depth Registers=true2026.04 | 23.7 | |
| FP16 referencek=64, Quantization=FP16, Depth Registers=true, Attention Sink=true2026.04 | 23.7 | |
| QuaRot (per-Linear) W4A4k=64, Quantization=W4A4, Depth Registers=true2026.04 | 25.4 | |
| QuaRot (per-Linear) W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.04 | 25.4 | |
| QuaRot (per-Linear) W4A4k=0, Quantization=W4A42026.04 | 25.5 | |
| QuaRot (full) W4A4k=0, Quantization=W4A42026.04 | 26 | |
| QuaRot (full) W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.04 | 26 | |
| QuaRot (full) W4A4k=64, Quantization=W4A4, Depth Registers=true2026.04 | 26.1 | |
| SmoothQuant W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.04 | 39.9 | |
| SmoothQuant W4A4k=0, Quantization=W4A42026.04 | 57.6 | |
| SmoothQuant W4A4k=64, Quantization=W4A4, Depth Registers=true2026.04 | 58.1 | |
| Naive RTN W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.04 | 119 | |
| Naive RTN W4A4k=64, Quantization=W4A4, Depth Registers=true2026.04 | 671 | |
| Naive RTN W4A4k=0, Quantization=W4A42026.04 | 1,727 |