List your company
ResearchFormerly WizWand

Language Modeling on FineWeb 10BT (train)

2.5966LossCompetitive MGR
2.5833442.6728222.76232.851778May 22, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Competitive MGRModel Scale=L, n=82026.052.5966
Independent MGRModel Scale=L, n=82026.052.5994
Competitive MGRModel Scale=L, n=42026.052.6001
Independent MGRModel Scale=L, n=42026.052.6006
Full AttnResModel Scale=L2026.052.6036
Block AttnResModel Scale=L, n=42026.052.6054
Pre NormModel Scale=L2026.052.6306
Competitive MGRModel Scale=M, n=42026.052.6889
Competitive MGRModel Scale=M, n=82026.052.6891
Independent MGRModel Scale=M, n=82026.052.6896
Independent MGRModel Scale=M, n=42026.052.6911
Block AttnResModel Scale=L, n=82026.052.6925
Full AttnResModel Scale=M2026.052.693
Block AttnResModel Scale=M, n=82026.052.6952
Block AttnResModel Scale=M, n=42026.052.6994
mHC-liteModel Scale=M, n=42026.052.7157
Pre NormModel Scale=M2026.052.7286
Competitive MGRModel Scale=S, n=82026.052.8869
Independent MGRModel Scale=S, n=82026.052.8877
Independent MGRModel Scale=S, n=42026.052.8887
Competitive MGRModel Scale=S, n=42026.052.8889
Full AttnResModel Scale=S2026.052.8911
Block AttnResModel Scale=S, n=82026.052.8921
Block AttnResModel Scale=S, n=42026.052.8951
mHC-liteModel Scale=S, n=42026.052.8992
Pre NormModel Scale=S2026.052.928