Language Modeling on SlimPajama large-scale (train)
2.129L(ψ)Coupled Adam
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Coupled AdamDataset size (D)=210B, Number of parameters (N)=2.6B2025.02 | 2.129 | 49.2 | 0.65 | 0.48 | 0.14 | 67 | 69 | |
| Standard AdamDataset size (D)=210B, Number of parameters (N)=2.6B2025.02 | 2.131 | 49 | 0.34 | 0.96 | 0.41 | 54 | -52 | |
| Standard AdamDataset size (D)=52B, Number of parameters (N)=2.6B2025.02 | 2.257 | 45.1 | 0.51 | 0.68 | 0.4 | 55 | -44 | |
| Coupled AdamDataset size (D)=52B, Number of parameters (N)=2.6B2025.02 | 2.273 | 44.1 | 0.86 | 0.1 | 0.06 | 66 | 74 | |
| Coupled AdamDataset size (D)=105B, Number of parameters (N)=1.3B2025.02 | 2.277 | 44.7 | 0.82 | 0.23 | 0.09 | 67 | 71 | |
| Standard AdamDataset size (D)=105B, Number of parameters (N)=1.3B2025.02 | 2.278 | 44.6 | 0.4 | 0.87 | 0.43 | 52 | -52 | |
| Standard AdamDataset size (D)=26B, Number of parameters (N)=1.3B2025.02 | 2.433 | 40.2 | 0.5 | 0.67 | 0.45 | 53 | -41 | |
| Coupled AdamDataset size (D)=26B, Number of parameters (N)=1.3B2025.02 | 2.448 | 39.6 | 0.96 | 0.05 | 0.03 | 66 | 74 |