Language Modeling on FineWeb-Edu 10K steps (train)
3.338Training LossDelta Block
Evaluation Results
| Method | Links | |
|---|---|---|
| Delta BlockScale=1044M, d=1280, L=36, Tok/s=86k, Mem (GB)=28.42026.05 | 3.338 | |
| Delta AttnResScale=1044M, d=1280, L=36, Tok/s=34k, Mem (GB)=77.72026.05 | 3.339 | |
| BaselineScale=1044M, d=1280, L=36, Tok/s=108k, Mem (GB)=22.52026.05 | 3.36 | |
| Delta AttnResScale=533M, d=1024, L=24, Tok/s=80k, Mem (GB)=33.62026.05 | 3.401 | |
| Delta BlockScale=533M, d=1024, L=24, Tok/s=198k, Mem (GB)=16.12026.05 | 3.405 | |
| Full AttnResScale=533M, d=1024, L=24, Tok/s=103k, Mem (GB)=24.82026.05 | 3.422 | |
| AttnResScale=533M, d=1024, L=24, Tok/s=174k, Mem (GB)=17.52026.05 | 3.423 | |
| BaselineScale=533M, d=1024, L=24, Tok/s=243k, Mem (GB)=13.92026.05 | 3.428 | |
| AttnResScale=1044M, d=1280, L=36, Tok/s=71k, Mem (GB)=31.62026.05 | 3.428 | |
| Full AttnResScale=1044M, d=1280, L=36, Tok/s=49k, Mem (GB)=52.02026.05 | 3.474 | |
| Delta AttnResScale=220M, d=768, L=12, Tok/s=246k, Mem (GB)=7.52026.05 | 3.563 | |
| Full AttnResScale=220M, d=768, L=12, Tok/s=285k, Mem (GB)=6.82026.05 | 3.577 | |
| Delta BlockScale=220M, d=768, L=12, Tok/s=306k, Mem (GB)=5.92026.05 | 3.577 | |
| AttnResScale=220M, d=768, L=12, Tok/s=294k, Mem (GB)=6.22026.05 | 3.586 | |
| BaselineScale=220M, d=768, L=12, Tok/s=289k, Mem (GB)=5.62026.05 | 3.613 |