Embedding Space Analysis on OpenWebText
0.98IsoCoupled Adam
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Coupled AdamDataset Size (D)=5B, Model Size (N)=355M2025.02 | 0.98 | 0.01 | 0.01 | 56 | 86 | 6.8 | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=760M2025.02 | 0.97 | 0.01 | 0.01 | 55 | 85 | 6.9 | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=355M2025.02 | 0.96 | 0.02 | 0.01 | 57 | 83 | 5.3 | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=760M2025.02 | 0.96 | 0.02 | 0.01 | 57 | 84 | 6.2 | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=355M2025.02 | 0.95 | 0.04 | 0.02 | 57 | 78 | 4.1 | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=125M2025.02 | 0.94 | 0.02 | 0.01 | 55 | 87 | 4.8 | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=760M2025.02 | 0.94 | 0.03 | 0.01 | 58 | 81 | 4.4 | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=125M2025.02 | 0.91 | 0.05 | 0.02 | 57 | 82 | 3.6 | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=125M2025.02 | 0.83 | 0.11 | 0.03 | 57 | 77 | 1.7 | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=355M2025.02 | 0.44 | 0.81 | 0.67 | 16 | -47 | 0.8 | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=760M2025.02 | 0.43 | 0.84 | 0.63 | 14 | -49 | 0.7 | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=760M2025.02 | 0.38 | 0.97 | 0.66 | 11 | -56 | 0.5 | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=355M2025.02 | 0.35 | 1.01 | 0.74 | 10 | -55 | 0.5 | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=125M2025.02 | 0.31 | 1.1 | 0.67 | 15 | -54 | 0.6 | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=760M2025.02 | 0.28 | 1.21 | 0.73 | 3 | -64 | 0.3 | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=355M2025.02 | 0.25 | 1.32 | 0.82 | 5 | -65 | 0.3 | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=125M2025.02 | 0.21 | 1.58 | 0.75 | 9 | -64 | 0.4 | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=125M2025.02 | 0.1 | 2.14 | 0.82 | 5 | -66 | 0.3 |