# Language Modeling on 12 diverse evaluation prompts (held-out)

Source: [Language Modeling on 12 diverse evaluation prompts (held-out)](<https://www.sota2.com/research/sota/language-modeling-on-12-diverse-evaluation-prompts-held-out>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-11T09:08:03.755Z

Evaluation of Language Modeling performance using Held-out Perplexity (PPL) on a specific set of 12 diverse, held-out evaluation prompts.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Language Modeling |
| Dataset (including any stated split) | 12 diverse evaluation prompts (held-out) |
| Metric | Held-out Perplexity |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 3 of 3 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Held-out Perplexity](<https://www.sota2.com/research/sota/language-modeling-on-12-diverse-evaluation-prompts-held-out.md?metric=Held-out%20Perplexity>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Held-out Perplexity | Result date | Report paper |
| --- | --- | --- | --- |
| BLOOM-1b7 (Stock) | 21.45 | Not recorded | 69b0f7312d603e6de3e45ff5 |
| Surgical Reinitialization (Pass 1) | 27.52 | Not recorded | 69b0f7312d603e6de3e45ff5 |
| Surgical Reinitialization (Pass 2) | 28.76 | Not recorded | 69b0f7312d603e6de3e45ff5 |

### BLOOM-1b7 (Stock)

Held-out Perplexity: 21.45. Result ID: 69b130ec6dbb7075b95bcccb.

[Reporting paper](<https://www.sota2.com/research/paper/69b0f7312d603e6de3e45ff5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6fdd1e209131cea47c06.md>)

Recorded evaluation configuration: {"Surgical Stage":"Stock","Params trained":"—"}. This record does not assert independent reproduction.

### Surgical Reinitialization (Pass 1)

Held-out Perplexity: 27.52. Result ID: 69b130ec6dbb7075b95bcccc.

[Reporting paper](<https://www.sota2.com/research/paper/69b0f7312d603e6de3e45ff5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69b0f7312d603e6de3e45ff5.md>)

Recorded evaluation configuration: {"Surgical Stage":"Pass 1","Epochs":"E3","Params trained":"302M (17.5%)"}. This record does not assert independent reproduction.

### Surgical Reinitialization (Pass 2)

Held-out Perplexity: 28.76. Result ID: 69b130ec6dbb7075b95bcccd.

[Reporting paper](<https://www.sota2.com/research/paper/69b0f7312d603e6de3e45ff5.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69b0f7312d603e6de3e45ff5.md>)

Recorded evaluation configuration: {"Surgical Stage":"Pass 2","Epochs":"E1","Params trained":"40M (2.3%)"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.