Language Modeling on FineWeb 10BT (train)
2.5966LossCompetitive MGR
Evaluation Results
| Method | Links | |
|---|---|---|
| Competitive MGRModel Scale=L, n=82026.05 | 2.5966 | |
| Independent MGRModel Scale=L, n=82026.05 | 2.5994 | |
| Competitive MGRModel Scale=L, n=42026.05 | 2.6001 | |
| Independent MGRModel Scale=L, n=42026.05 | 2.6006 | |
| Full AttnResModel Scale=L2026.05 | 2.6036 | |
| Block AttnResModel Scale=L, n=42026.05 | 2.6054 | |
| Pre NormModel Scale=L2026.05 | 2.6306 | |
| Competitive MGRModel Scale=M, n=42026.05 | 2.6889 | |
| Competitive MGRModel Scale=M, n=82026.05 | 2.6891 | |
| Independent MGRModel Scale=M, n=82026.05 | 2.6896 | |
| Independent MGRModel Scale=M, n=42026.05 | 2.6911 | |
| Block AttnResModel Scale=L, n=82026.05 | 2.6925 | |
| Full AttnResModel Scale=M2026.05 | 2.693 | |
| Block AttnResModel Scale=M, n=82026.05 | 2.6952 | |
| Block AttnResModel Scale=M, n=42026.05 | 2.6994 | |
| mHC-liteModel Scale=M, n=42026.05 | 2.7157 | |
| Pre NormModel Scale=M2026.05 | 2.7286 | |
| Competitive MGRModel Scale=S, n=82026.05 | 2.8869 | |
| Independent MGRModel Scale=S, n=82026.05 | 2.8877 | |
| Independent MGRModel Scale=S, n=42026.05 | 2.8887 | |
| Competitive MGRModel Scale=S, n=42026.05 | 2.8889 | |
| Full AttnResModel Scale=S2026.05 | 2.8911 | |
| Block AttnResModel Scale=S, n=82026.05 | 2.8921 | |
| Block AttnResModel Scale=S, n=42026.05 | 2.8951 | |
| mHC-liteModel Scale=S, n=42026.05 | 2.8992 | |
| Pre NormModel Scale=S2026.05 | 2.928 |