List your company

Language Modeling on The Pile (Context Scaling PPL)

28.18PPL (No ft)LLM
27.842830.118932.39534.6711Feb 2, 2024

Evaluation results

MethodLinks
LLMPretraining budget (GPUh)=200, Training steps=217k, Number of GPUs=82024.0228.1815.6214.0310.81
LLMPretraining budget (GPUh)=600, Training steps=651k, Number of GPUs=82024.0228.6214.512.6410.07
LLMPretraining budget (GPUh)=400, Training steps=434k, Number of GPUs=82024.0228.9815.0313.0510.28
LLMPretraining budget (GPUh)=100, Training steps=108k, Number of GPUs=82024.0229.2217.1115.4911.55
SLM-mixPretraining budget (GPUh)=600, Training steps=1000k, Number of GPUs=162024.0231.0317.1814.2111.73
SLM-mixPretraining budget (GPUh)=400, Training steps=1000k, Number of GPUs=162024.0231.0417.5614.4211.84
SLM-mixPretraining budget (GPUh)=200, Training steps=928k, Number of GPUs=162024.0231.9418.4814.9812.15
SLM-pnPretraining budget (GPUh)=200, Training steps=390k, Number of GPUs=82024.0232.317.7614.9512.13
SLM-pnPretraining budget (GPUh)=600, Training steps=1170k, Number of GPUs=82024.0232.5316.9514.2911.74
SLM-pnPretraining budget (GPUh)=400, Training steps=780k, Number of GPUs=82024.0232.5417.1714.4811.86
SLM-pnPretraining budget (GPUh)=100, Training steps=195k, Number of GPUs=82024.0233.4418.5715.5812.53
SLMPretraining budget (GPUh)=100, Training steps=798k, Number of GPUs=82024.0233.7419.3115.6112.37
SLM-mixPretraining budget (GPUh)=100, Training steps=464k, Number of GPUs=162024.0234.3519.8215.8212.62
SLMPretraining budget (GPUh)=200, Training steps=1597k, Number of GPUs=82024.0234.4318.5815.1212.09
SLMPretraining budget (GPUh)=400, Training steps=3195k, Number of GPUs=82024.0236.6118.2214.812