List your company

Language Modeling on FineWeb-Edu 300M×5B (val)

23.6PPLFP16 reference
-44.536415.382875.31,335.218Apr 20, 2026

Evaluation results

MethodLinks
FP16 referencek=0, Quantization=FP162026.0423.6
FP16 referencek=64, Quantization=FP16, Depth Registers=true2026.0423.7
FP16 referencek=64, Quantization=FP16, Depth Registers=true, Attention Sink=true2026.0423.7
QuaRot (per-Linear) W4A4k=64, Quantization=W4A4, Depth Registers=true2026.0425.4
QuaRot (per-Linear) W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.0425.4
QuaRot (per-Linear) W4A4k=0, Quantization=W4A42026.0425.5
QuaRot (full) W4A4k=0, Quantization=W4A42026.0426
QuaRot (full) W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.0426
QuaRot (full) W4A4k=64, Quantization=W4A4, Depth Registers=true2026.0426.1
SmoothQuant W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.0439.9
SmoothQuant W4A4k=0, Quantization=W4A42026.0457.6
SmoothQuant W4A4k=64, Quantization=W4A4, Depth Registers=true2026.0458.1
Naive RTN W4A4k=64, Quantization=W4A4, Depth Registers=true, Attention Sink=true2026.04119
Naive RTN W4A4k=64, Quantization=W4A4, Depth Registers=true2026.04671
Naive RTN W4A4k=0, Quantization=W4A42026.041,727