Language Modeling on Zyda 2
2.286LossCCGQA
Evaluation results
| Method | Links | |
|---|---|---|
| CCGQAParameters=1B, Training Tokens=300B, Architecture=Dense2025.10 | 2.286 | |
| MHAParameters=1B, Training Tokens=300B, Architecture=Dense2025.10 | 2.297 | |
| GQAParameters=1B, Training Tokens=300B, Architecture=Dense2025.10 | 2.297 | |
| CCAParameters=1B, Training Tokens=300B, Architecture=Dense2025.10 | 2.307 | |
| MLAParameters=1B, Training Tokens=300B, Architecture=Dense2025.10 | 2.321 |