List your company
ResearchFormerly WizWand

Language Modeling on FineWeb-Edu 10K steps (train)

3.338Training LossDelta Block
3.3273.401253.47553.54975May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Delta BlockScale=1044M, d=1280, L=36, Tok/s=86k, Mem (GB)=28.42026.053.338
Delta AttnResScale=1044M, d=1280, L=36, Tok/s=34k, Mem (GB)=77.72026.053.339
BaselineScale=1044M, d=1280, L=36, Tok/s=108k, Mem (GB)=22.52026.053.36
Delta AttnResScale=533M, d=1024, L=24, Tok/s=80k, Mem (GB)=33.62026.053.401
Delta BlockScale=533M, d=1024, L=24, Tok/s=198k, Mem (GB)=16.12026.053.405
Full AttnResScale=533M, d=1024, L=24, Tok/s=103k, Mem (GB)=24.82026.053.422
AttnResScale=533M, d=1024, L=24, Tok/s=174k, Mem (GB)=17.52026.053.423
BaselineScale=533M, d=1024, L=24, Tok/s=243k, Mem (GB)=13.92026.053.428
AttnResScale=1044M, d=1280, L=36, Tok/s=71k, Mem (GB)=31.62026.053.428
Full AttnResScale=1044M, d=1280, L=36, Tok/s=49k, Mem (GB)=52.02026.053.474
Delta AttnResScale=220M, d=768, L=12, Tok/s=246k, Mem (GB)=7.52026.053.563
Full AttnResScale=220M, d=768, L=12, Tok/s=285k, Mem (GB)=6.82026.053.577
Delta BlockScale=220M, d=768, L=12, Tok/s=306k, Mem (GB)=5.92026.053.577
AttnResScale=220M, d=768, L=12, Tok/s=294k, Mem (GB)=6.22026.053.586
BaselineScale=220M, d=768, L=12, Tok/s=289k, Mem (GB)=5.62026.053.613