Language Modeling on The Pile (Context Scaling PPL)
28.18PPL (No ft)LLM
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| LLMPretraining budget (GPUh)=200, Training steps=217k, Number of GPUs=82024.02 | 28.18 | 15.62 | 14.03 | 10.81 | |
| LLMPretraining budget (GPUh)=600, Training steps=651k, Number of GPUs=82024.02 | 28.62 | 14.5 | 12.64 | 10.07 | |
| LLMPretraining budget (GPUh)=400, Training steps=434k, Number of GPUs=82024.02 | 28.98 | 15.03 | 13.05 | 10.28 | |
| LLMPretraining budget (GPUh)=100, Training steps=108k, Number of GPUs=82024.02 | 29.22 | 17.11 | 15.49 | 11.55 | |
| SLM-mixPretraining budget (GPUh)=600, Training steps=1000k, Number of GPUs=162024.02 | 31.03 | 17.18 | 14.21 | 11.73 | |
| SLM-mixPretraining budget (GPUh)=400, Training steps=1000k, Number of GPUs=162024.02 | 31.04 | 17.56 | 14.42 | 11.84 | |
| SLM-mixPretraining budget (GPUh)=200, Training steps=928k, Number of GPUs=162024.02 | 31.94 | 18.48 | 14.98 | 12.15 | |
| SLM-pnPretraining budget (GPUh)=200, Training steps=390k, Number of GPUs=82024.02 | 32.3 | 17.76 | 14.95 | 12.13 | |
| SLM-pnPretraining budget (GPUh)=600, Training steps=1170k, Number of GPUs=82024.02 | 32.53 | 16.95 | 14.29 | 11.74 | |
| SLM-pnPretraining budget (GPUh)=400, Training steps=780k, Number of GPUs=82024.02 | 32.54 | 17.17 | 14.48 | 11.86 | |
| SLM-pnPretraining budget (GPUh)=100, Training steps=195k, Number of GPUs=82024.02 | 33.44 | 18.57 | 15.58 | 12.53 | |
| SLMPretraining budget (GPUh)=100, Training steps=798k, Number of GPUs=82024.02 | 33.74 | 19.31 | 15.61 | 12.37 | |
| SLM-mixPretraining budget (GPUh)=100, Training steps=464k, Number of GPUs=162024.02 | 34.35 | 19.82 | 15.82 | 12.62 | |
| SLMPretraining budget (GPUh)=200, Training steps=1597k, Number of GPUs=82024.02 | 34.43 | 18.58 | 15.12 | 12.09 | |
| SLMPretraining budget (GPUh)=400, Training steps=3195k, Number of GPUs=82024.02 | 36.61 | 18.22 | 14.8 | 12 |