Language Modeling on FineWeb-Edu (Precision-Specific PPL and Training Throughput)
8.49Perplexity (BF16)Hyperloop
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| HyperloopDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=990.8 M, Loop Architecture=[4L → 10L (×3) → 4L]2026.04 | 8.49 | 8.59 | 180,000 | |
| mHCDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=2033.0 M2026.04 | 8.57 | 8.62 | 109,000 | |
| TransformerDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=2018.0 M2026.04 | 8.6 | 8.71 | 181,000 | |
| LoopedDim=2048, Unrolled Depth=38, Train Tokens=100B, Params=990.5 M, Loop Architecture=[4L → 10L (×3) → 4L]2026.04 | 8.68 | 8.97 | 183,000 | |
| HyperloopDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=579.7 M, Loop Architecture=[3L → 4L (×3) → 3L]2026.04 | 9.65 | 9.81 | 354,000 | |
| LoopedDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=579.4 M, Loop Architecture=[3L → 4L (×3) → 3L]2026.04 | 10.02 | 10.24 | 367,000 | |
| mHCDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=997.5 M2026.04 | 10.07 | 10.16 | 237,000 | |
| TransformerDim=2048, Unrolled Depth=18, Train Tokens=50B, Params=990.5 M2026.04 | 10.19 | 10.27 | 367,000 | |
| HyperloopDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=135.7 M, Loop Architecture=[2L → 4L (×3) → 2L]2026.04 | 14.4 | 14.68 | 750,000 | |
| mHCDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=241.0 M2026.04 | 14.55 | 14.73 | 514,000 | |
| TransformerDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=238.0 M2026.04 | 14.65 | 14.85 | 786,000 | |
| LoopedDim=1024, Unrolled Depth=16, Train Tokens=12.5B, Params=135.5 M, Loop Architecture=[2L → 4L (×3) → 2L]2026.04 | 14.85 | 15.18 | 786,000 |