Language Modeling on (val) using Validation Loss
2.524Validation LossPowerStep
Evaluation Results
| Method | Links | |
|---|---|---|
| PowerStepModel=Qwen3-235B-A22B, Precision=int82026.05 | 2.524 | |
| AdamWModel=Qwen3-235B-A22B, Precision=fp322026.05 | 2.525 | |
| AdamWModel=Qwen3-32B, Precision=fp322026.05 | 2.572 | |
| PowerStepModel=Qwen3-32B, Precision=int82026.05 | 2.582 | |
| PowerStepModel=DeepSeek-V2-Lite (16B), Precision=int82026.05 | 2.618 | |
| AdamWModel=Qwen3-30B-A3B, Precision=fp322026.05 | 2.62 | |
| AdamWModel=DeepSeek-V2-Lite (16B), Precision=fp322026.05 | 2.623 | |
| PowerStepModel=Qwen3-30B-A3B, Precision=int82026.05 | 2.624 | |
| AdamWModel=Qwen3-8B, Optimizer state=fp322026.05 | 2.66 | |
| AdamSModel=Qwen3-4B, Optimizer state=fp322026.05 | 2.672 | |
| SignSGDModel=Qwen3-4B, Optimizer state=fp322026.05 | 2.68 | |
| PowerStepModel=Qwen3-8B, Optimizer state=fp322026.05 | 2.688 | |
| AdamWModel=Qwen3-4B, Optimizer state=fp322026.05 | 2.69 | |
| AdamWModel=GPT-2-Medium (350M), Optimizer state=fp322026.05 | 2.711 | |
| pbSGDMModel=Qwen3-4B, Optimizer state=fp322026.05 | 2.712 | |
| PowerStepModel=Qwen3-4B, Optimizer state=fp322026.05 | 2.712 | |
| PowerStepModel=GPT-2-Medium (350M), Optimizer state=fp322026.05 | 2.717 | |
| AdamSModel=GPT-2-Medium (350M), Optimizer state=fp322026.05 | 2.725 | |
| SignSGDModel=GPT-2-Medium (350M), Optimizer state=fp322026.05 | 2.736 | |
| AdamWModel=Qwen3-1.7B, Optimizer state=fp322026.05 | 2.78 | |
| AdamSModel=Qwen3-1.7B, Optimizer state=fp322026.05 | 2.782 | |
| SignSGDModel=Qwen3-1.7B, Optimizer state=fp322026.05 | 2.794 | |
| PowerStepModel=Qwen3-1.7B, Optimizer state=fp322026.05 | 2.799 | |
| pbSGDMModel=Qwen3-8B, Optimizer state=fp322026.05 | 2.853 | |
| SignSGDModel=Qwen3-0.6B, Optimizer state=fp322026.05 | 2.897 | |
| AdamWModel=Qwen3-0.6B, Optimizer state=fp322026.05 | 2.912 | |
| AdamSModel=Qwen3-0.6B, Optimizer state=fp322026.05 | 2.912 | |
| PowerStepModel=Qwen3-0.6B, Optimizer state=fp322026.05 | 2.923 | |
| SignSGDModel=GPT-2-Small (124M), Optimizer state=fp322026.05 | 2.942 | |
| PowerStepModel=GPT-2-Small (124M), Optimizer state=fp322026.05 | 2.949 | |
| AdamWModel=GPT-2-Small (124M), Optimizer state=fp322026.05 | 2.95 | |
| AdamSModel=GPT-2-Small (124M), Optimizer state=fp322026.05 | 2.951 | |
| pbSGDMModel=Qwen3-1.7B, Optimizer state=fp322026.05 | 3.034 | |
| pbSGDMModel=GPT-2-Medium (350M), Optimizer state=fp322026.05 | 3.11 | |
| pbSGDMModel=Qwen3-0.6B, Optimizer state=fp322026.05 | 3.198 | |
| pbSGDMModel=GPT-2-Small (124M), Optimizer state=fp322026.05 | 3.377 | |
| AdamSModel=Qwen3-8B, Optimizer state=fp322026.05 | 4.689 | |
| SignSGDModel=Qwen3-8B, Optimizer state=fp322026.05 | 5.045 |