Language Modeling on FineWeb-Edu 10K steps (val)
3.372Validation LossDelta AttnRes
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Delta AttnResScale=1044M, d=1280, L=36, Tok/s=34k, Mem (GB)=77.72026.05 | 3.372 | 29.13 | |
| Delta BlockScale=1044M, d=1280, L=36, Tok/s=86k, Mem (GB)=28.42026.05 | 3.374 | 29.19 | |
| BaselineScale=1044M, d=1280, L=36, Tok/s=108k, Mem (GB)=22.52026.05 | 3.391 | 29.7 | |
| Delta AttnResScale=533M, d=1024, L=24, Tok/s=80k, Mem (GB)=33.62026.05 | 3.436 | 31.05 | |
| Delta BlockScale=533M, d=1024, L=24, Tok/s=198k, Mem (GB)=16.12026.05 | 3.439 | 31.16 | |
| Full AttnResScale=533M, d=1024, L=24, Tok/s=103k, Mem (GB)=24.82026.05 | 3.456 | 31.68 | |
| AttnResScale=533M, d=1024, L=24, Tok/s=174k, Mem (GB)=17.52026.05 | 3.458 | 31.75 | |
| AttnResScale=1044M, d=1280, L=36, Tok/s=71k, Mem (GB)=31.62026.05 | 3.458 | 31.76 | |
| BaselineScale=533M, d=1024, L=24, Tok/s=243k, Mem (GB)=13.92026.05 | 3.466 | 32 | |
| Full AttnResScale=1044M, d=1280, L=36, Tok/s=49k, Mem (GB)=52.02026.05 | 3.508 | 33.36 | |
| Delta AttnResScale=220M, d=768, L=12, Tok/s=246k, Mem (GB)=7.52026.05 | 3.606 | 36.83 | |
| Delta BlockScale=220M, d=768, L=12, Tok/s=306k, Mem (GB)=5.92026.05 | 3.613 | 37.08 | |
| Full AttnResScale=220M, d=768, L=12, Tok/s=285k, Mem (GB)=6.82026.05 | 3.619 | 37.3 | |
| AttnResScale=220M, d=768, L=12, Tok/s=294k, Mem (GB)=6.22026.05 | 3.622 | 37.39 | |
| BaselineScale=220M, d=768, L=12, Tok/s=289k, Mem (GB)=5.62026.05 | 3.656 | 38.71 |