LLM Pretraining on FineWeb-Edu (train)
2.964Training LossmHC
Evaluation results
| Method | Links | |
|---|---|---|
| mHCD (Number of transformer blocks)=12, n (Number of residual streams)=4, ∆ Params (K)=18442026.01 | 2.964 | |
| KromHCD (Number of transformer blocks)=12, n (Number of residual streams)=4, ∆ Params (K)=9592026.01 | 2.966 | |
| ResidualD (Number of transformer blocks)=12, n (Number of residual streams)=42026.01 | 2.971 | |
| mHC-liteD (Number of transformer blocks)=12, n (Number of residual streams)=4, ∆ Params (K)=24332026.01 | 2.972 | |
| mHC-liteD (Number of transformer blocks)=6, n (Number of residual streams)=4, ∆ Params (K)=6092026.01 | 3.484 | |
| KromHCD (Number of transformer blocks)=6, n (Number of residual streams)=4, ∆ Params (K)=2402026.01 | 3.488 | |
| ResidualD (Number of transformer blocks)=6, n (Number of residual streams)=42026.01 | 3.49 | |
| mHCD (Number of transformer blocks)=6, n (Number of residual streams)=4, ∆ Params (K)=4622026.01 | 3.493 |