Supervised Fine-Tuning on SFT (val)
1.3519Validation LossEmber
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| EmberModel=pythia-2.8b, Params (M)=2800, Step=250, Optimizer State Precision=fp322026.07 | 1.3519 | 0.0008 | 413,000 | |
| AdamWModel=pythia-2.8b, Params (M)=2800, Step=250, Optimizer State Precision=fp322026.07 | 1.3527 | — | 2,060 | |
| EmberModel=pythia-1.4b, Params (M)=1400, Step=250, Optimizer State Precision=fp322026.07 | 1.4306 | 0.0003 | 409,000 | |
| AdamWModel=pythia-1.4b, Params (M)=1400, Step=250, Optimizer State Precision=fp322026.07 | 1.4309 | — | 1,650 | |
| EmberModel=gpt2-large, Params (M)=774, Step=250, Optimizer State Precision=fp322026.07 | 1.5888 | 0.0012 | 201,000 | |
| AdamWModel=gpt2-large, Params (M)=774, Step=250, Optimizer State Precision=fp322026.07 | 1.59 | — | 515,000 | |
| EmberModel=gpt2-medium, Params (M)=350, Step=250, Optimizer State Precision=fp322026.07 | 1.743 | 0.0037 | 200,000 | |
| AdamWModel=gpt2-medium, Params (M)=350, Step=250, Optimizer State Precision=fp322026.07 | 1.7467 | — | 412,000 | |
| EmberModel=gpt2, Params (M)=124, Step=250, Optimizer State Precision=fp322026.07 | 2.0381 | 0.0075 | 199,000 | |
| AdamWModel=gpt2, Params (M)=124, Step=250, Optimizer State Precision=fp322026.07 | 2.0456 | — | 309,000 | |
| EmberModel=distilgpt2, Params (M)=82, Step=250, Optimizer State Precision=fp322026.07 | 2.3578 | 0.0062 | 199,000 | |
| AdamWModel=distilgpt2, Params (M)=82, Step=250, Optimizer State Precision=fp322026.07 | 2.364 | — | 309,000 | |
| AdamWModel=pythia-160m, Params (M)=160, Step=250, Optimizer State Precision=fp322026.07 | 2.4782 | — | 618,000 | |
| EmberModel=pythia-160m, Params (M)=160, Step=250, Optimizer State Precision=fp322026.07 | 2.5136 | 0.0354 | 399,000 |