Language Modeling on GPT-2 Pre-training (val)
2.493Validation LossDense
Evaluation Results
| Method | Links | |
|---|---|---|
| DenseParams=774M, Pre-train=Dense2024.09 | 2.493 | |
| S-STEParams=774M, Pre-train=S-STE2024.09 | 2.547 | |
| T-SR-STE+DFParams=774M, Pre-train=T-SR-STE+DF [20]2024.09 | 2.564 | |
| DenseParams=350M, Pre-train=Dense2024.09 | 2.618 | |
| T-SR-STE+DFParams=350M, Pre-train=T-SR-STE+DF [20]2024.09 | 2.688 | |
| SR-STEParams=350M, Pre-train=SR-STE2024.09 | 2.69 | |
| S-STEParams=350M, Pre-train=S-STE2024.09 | 2.713 | |
| T-SR-STEParams=350M, Pre-train=T-SR-STE2024.09 | 2.718 | |
| DenseParams=124M, Pre-train=Dense2024.09 | 2.907 | |
| NAMO-DModel=GPT-2 (355M), Steps=10K2026.02 | 2.9507 | |
| NAMOModel=GPT-2 (355M), Steps=10K2026.02 | 2.9516 | |
| T-SR-STE+DFParams=124M, Pre-train=T-SR-STE+DF [20]2024.09 | 2.952 | |
| MuonModel=GPT-2 (355M), Steps=10K2026.02 | 2.9684 | |
| SR-STEParams=124M, Pre-train=SR-STE2024.09 | 2.982 | |
| S-STEParams=124M, Pre-train=S-STE2024.09 | 2.984 | |
| AdamWModel=GPT-2 (355M), Steps=10K2026.02 | 2.9914 | |
| NAMO-DModel=GPT-2 (124M), Steps=50K2026.02 | 3.0246 | |
| NAMOModel=GPT-2 (124M), Steps=50K2026.02 | 3.0351 | |
| MuonModel=GPT-2 (124M), Steps=50K2026.02 | 3.0435 | |
| AdamWModel=GPT-2 (124M), Steps=50K2026.02 | 3.0643 | |
| T-SR-STEParams=124M, Pre-train=T-SR-STE2024.09 | 3.076 |