List your company
ResearchFormerly WizWand

Language Modeling on FineWeb-Edu (Precision-Specific PPL and Training Throughput)

8.49Perplexity (BF16)Hyperloop
8.23569.952811.6713.3872Apr 23, 2026
Updated 6mo ago

Evaluation results

MethodLinks
HyperloopDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=990.8 M, Loop Architecture=[4L → 10L (×3) → 4L]2026.048.498.59180,000
mHCDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=2033.0 M2026.048.578.62109,000
TransformerDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=2018.0 M2026.048.68.71181,000
LoopedDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=990.5 M, Loop Architecture=[4L → 10L (×3) → 4L]2026.048.688.97183,000
HyperloopDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=579.7 M, Loop Architecture=[3L → 4L (×3) → 3L]2026.049.659.81354,000
LoopedDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=579.4 M, Loop Architecture=[3L → 4L (×3) → 3L]2026.0410.0210.24367,000
mHCDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=997.5 M2026.0410.0710.16237,000
TransformerDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=990.5 M2026.0410.1910.27367,000
HyperloopDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=135.7 M, Loop Architecture=[2L → 4L (×3) → 2L]2026.0414.414.68750,000
mHCDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=241.0 M2026.0414.5514.73514,000
TransformerDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=238.0 M2026.0414.6514.85786,000
LoopedDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=135.5 M, Loop Architecture=[2L → 4L (×3) → 2L]2026.0414.8515.18786,000