Language Modeling on Generic dataset
14.09PPLLLM
Evaluation results
| Method | Links | |
|---|---|---|
| LLMPretraining budget (GPUh)=600, Training steps=651k, Number of GPUs=82024.02 | 14.09 | |
| LLMPretraining budget (GPUh)=400, Training steps=434k, Number of GPUs=82024.02 | 14.54 | |
| LLMPretraining budget (GPUh)=200, Training steps=217k, Number of GPUs=82024.02 | 15.58 | |
| SLM-mixPretraining budget (GPUh)=400, Training steps=1000k, Number of GPUs=162024.02 | 15.82 | |
| SLM-mixPretraining budget (GPUh)=600, Training steps=1000k, Number of GPUs=162024.02 | 15.82 | |
| SLM-mixPretraining budget (GPUh)=200, Training steps=928k, Number of GPUs=162024.02 | 15.92 | |
| SLM-pnPretraining budget (GPUh)=600, Training steps=1170k, Number of GPUs=82024.02 | 16.53 | |
| SLM-pnPretraining budget (GPUh)=400, Training steps=780k, Number of GPUs=82024.02 | 16.9 | |
| LLMPretraining budget (GPUh)=100, Training steps=108k, Number of GPUs=82024.02 | 17 | |
| SLM-mixPretraining budget (GPUh)=100, Training steps=464k, Number of GPUs=162024.02 | 17.13 | |
| SLM-pnPretraining budget (GPUh)=200, Training steps=390k, Number of GPUs=82024.02 | 17.74 | |
| SLM-pnPretraining budget (GPUh)=100, Training steps=195k, Number of GPUs=82024.02 | 18.9 | |
| SLMPretraining budget (GPUh)=400, Training steps=3195k, Number of GPUs=82024.02 | 19.17 | |
| SLMPretraining budget (GPUh)=200, Training steps=1597k, Number of GPUs=82024.02 | 19.71 | |
| SLMPretraining budget (GPUh)=100, Training steps=798k, Number of GPUs=82024.02 | 20.51 |