LLM Pretraining on FineWeb-Edu (val)
0.861BPBmHC
Evaluation results
| Method | Links | |
|---|---|---|
| mHCD (Number of transformer blocks)=12, n (Number of residual streams)=4, ∆ Params (K)=18442026.01 | 0.861 | |
| KromHCD (Number of transformer blocks)=12, n (Number of residual streams)=4, ∆ Params (K)=9592026.01 | 0.862 | |
| ResidualD (Number of transformer blocks)=12, n (Number of residual streams)=42026.01 | 0.864 | |
| mHC-liteD (Number of transformer blocks)=12, n (Number of residual streams)=4, ∆ Params (K)=24332026.01 | 0.864 | |
| mHCD (Number of transformer blocks)=6, n (Number of residual streams)=4, ∆ Params (K)=4622026.01 | 1.042 | |
| mHC-liteD (Number of transformer blocks)=6, n (Number of residual streams)=4, ∆ Params (K)=6092026.01 | 1.045 | |
| ResidualD (Number of transformer blocks)=6, n (Number of residual streams)=42026.01 | 1.047 | |
| KromHCD (Number of transformer blocks)=6, n (Number of residual streams)=4, ∆ Params (K)=2402026.01 | 1.047 |