# Language Modeling on The Pile (Context Scaling PPL)

Source: [Language Modeling on The Pile (Context Scaling PPL)](<https://www.sota2.com/research/sota/language-modeling-on-the-pile-context-scaling-ppl>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:55:11.903Z

Evaluation of language modeling performance on The Pile dataset, reporting Perplexity measured under specific context window conditions (e.g., No ft, 1M, 8M, 64M tokens).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Language Modeling |
| Dataset (including any stated split) | Specialized dataset Pile |
| Metric | PPL (No ft) |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [PPL (No ft)](<https://www.sota2.com/research/sota/language-modeling-on-the-pile-context-scaling-ppl.md?metric=PPL%20(No%20ft)>): Lower is better; unit: SCALAR.

- [PPL (1M Context)](<https://www.sota2.com/research/sota/language-modeling-on-the-pile-context-scaling-ppl.md?metric=PPL%20(1M%20Context)>): Lower is better; unit: SCALAR.

- [PPL (8M Context)](<https://www.sota2.com/research/sota/language-modeling-on-the-pile-context-scaling-ppl.md?metric=PPL%20(8M%20Context)>): Lower is better; unit: SCALAR.

- [PPL (64M Context)](<https://www.sota2.com/research/sota/language-modeling-on-the-pile-context-scaling-ppl.md?metric=PPL%20(64M%20Context)>): Lower is better; unit: SCALAR.

## Recorded results

| Method | PPL (No ft) | Result date | Report paper |
| --- | --- | --- | --- |
| LLM | 28.18 | Not recorded | 68fa757a1e209131cea6839d |
| LLM | 28.62 | Not recorded | 68fa757a1e209131cea6839d |
| LLM | 28.98 | Not recorded | 68fa757a1e209131cea6839d |
| LLM | 29.22 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-mix | 31.03 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-mix | 31.04 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-mix | 31.94 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-pn | 32.3 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-pn | 32.53 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-pn | 32.54 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-pn | 33.44 | Not recorded | 68fa757a1e209131cea6839d |
| SLM | 33.74 | Not recorded | 68fa757a1e209131cea6839d |
| SLM-mix | 34.35 | Not recorded | 68fa757a1e209131cea6839d |
| SLM | 34.43 | Not recorded | 68fa757a1e209131cea6839d |
| SLM | 36.61 | Not recorded | 68fa757a1e209131cea6839d |

### LLM

PPL (No ft): 28.18. Result ID: 6993c9093a6c3148491882d0.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"200","Training steps":"217k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### LLM

PPL (No ft): 28.62. Result ID: 6993c9093a6c3148491882d7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"600","Training steps":"651k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### LLM

PPL (No ft): 28.98. Result ID: 6993c9093a6c3148491882d4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"400","Training steps":"434k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### LLM

PPL (No ft): 29.22. Result ID: 6993c9093a6c3148491882cc.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"100","Training steps":"108k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM-mix

PPL (No ft): 31.03. Result ID: 6993c9093a6c3148491882d5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa634f1e209131ce9fe07f.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"600","Training steps":"1000k","Number of GPUs":"16"}. This record does not assert independent reproduction.

### SLM-mix

PPL (No ft): 31.04. Result ID: 6993c9093a6c3148491882d2.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa634f1e209131ce9fe07f.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"400","Training steps":"1000k","Number of GPUs":"16"}. This record does not assert independent reproduction.

### SLM-mix

PPL (No ft): 31.94. Result ID: 6993c9093a6c3148491882ce.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa634f1e209131ce9fe07f.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"200","Training steps":"928k","Number of GPUs":"16"}. This record does not assert independent reproduction.

### SLM-pn

PPL (No ft): 32.3. Result ID: 6993c9093a6c3148491882cf.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"200","Training steps":"390k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM-pn

PPL (No ft): 32.53. Result ID: 6993c9093a6c3148491882d6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"600","Training steps":"1170k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM-pn

PPL (No ft): 32.54. Result ID: 6993c9093a6c3148491882d3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"400","Training steps":"780k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM-pn

PPL (No ft): 33.44. Result ID: 6993c9093a6c3148491882cb.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"100","Training steps":"195k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM

PPL (No ft): 33.74. Result ID: 6993c9093a6c3148491882c9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"100","Training steps":"798k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM-mix

PPL (No ft): 34.35. Result ID: 6993c9093a6c3148491882ca.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa634f1e209131ce9fe07f.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"100","Training steps":"464k","Number of GPUs":"16"}. This record does not assert independent reproduction.

### SLM

PPL (No ft): 34.43. Result ID: 6993c9093a6c3148491882cd.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"200","Training steps":"1597k","Number of GPUs":"8"}. This record does not assert independent reproduction.

### SLM

PPL (No ft): 36.61. Result ID: 6993c9093a6c3148491882d1.

[Reporting paper](<https://www.sota2.com/research/paper/68fa757a1e209131cea6839d.md>)

Recorded evaluation configuration: {"Pretraining budget (GPUh)":"400","Training steps":"3195k","Number of GPUs":"8"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
