# Pretraining

Source: [Pretraining](<https://www.sota2.com/research/task/pretraining>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.



## Benchmarks

Page 1 of 1. This is an index page, not the full collection.

- [LLM Pretraining on C4](<https://www.sota2.com/research/sota/llm-pretraining-on-c4.md>): LLM Pretraining; C4.

- [LLM Pretraining on C4 (Performance and Memory)](<https://www.sota2.com/research/sota/llm-pretraining-on-c4-performance-and-memory.md>): LLM Pretraining; C4.

- [Speech Pretraining on LibriSpeech (val)](<https://www.sota2.com/research/sota/speech-pretraining-on-librispeech-val.md>): Speech pretraining; LibriSpeech (val).

- [Tabular Foundation Model Pretraining on TabArena 38 classification tasks subsampled 2025 (5-fold CV)](<https://www.sota2.com/research/sota/tabular-foundation-model-pretraining-on-tabarena-38-classification-tasks-subsampled-2025-5-fold-cv.md>): Tabular Foundation Model Pretraining; TabArena 38 classification tasks subsampled 2025 (5-fold CV).

- [LLM Pretraining on FineWeb-Edu (val)](<https://www.sota2.com/research/sota/llm-pretraining-on-fineweb-edu-val.md>): LLM Pretraining; FineWeb-Edu (val).

- [LLM Pretraining on FineWeb-Edu (train)](<https://www.sota2.com/research/sota/llm-pretraining-on-fineweb-edu-train.md>): LLM Pretraining; FineWeb-Edu (train).

- [Self-supervised pretraining on Libri-Light 6k](<https://www.sota2.com/research/sota/self-supervised-pretraining-on-libri-light-6k.md>): Self-supervised pretraining; Libri-Light 6k.

- [LLM Pretraining on C4 (Final val)](<https://www.sota2.com/research/sota/llm-pretraining-on-c4-final-val.md>): LLM pretraining; C4 (Final val).

- [LLM Pretraining on PleIAs SYNTH (val)](<https://www.sota2.com/research/sota/llm-pretraining-on-pleias-synth-val.md>): LLM Pretraining; PleIAs SYNTH (val).

- [Pretraining on Llama pretraining 124M](<https://www.sota2.com/research/sota/pretraining-on-llama-pretraining-124m.md>): Pretraining; Llama pretraining 124M.

- [Continued Pretraining on SciERC (test)](<https://www.sota2.com/research/sota/continued-pretraining-on-scierc-test.md>): Continued Pretraining; SciERC (test).

- [Continued Pretraining on ACL-ARC (test)](<https://www.sota2.com/research/sota/continued-pretraining-on-acl-arc-test.md>): Continued Pretraining; ACL-ARC (test).

- [Continued Pretraining on HyperPartisan (test)](<https://www.sota2.com/research/sota/continued-pretraining-on-hyperpartisan-test.md>): Continued Pretraining; HyperPartisan (test).

- [Continued Pretraining on ChemProt (test)](<https://www.sota2.com/research/sota/continued-pretraining-on-chemprot-test.md>): Continued Pretraining; ChemProt (test).

- [Pretraining Data Mixture Estimation on OLMo Pretraining Mixture (Temporal held-out)](<https://www.sota2.com/research/sota/pretraining-data-mixture-estimation-on-olmo-pretraining-mixture-temporal-held-out.md>): Pretraining Data Mixture Estimation; OLMo Pretraining Mixture (Temporal held-out).

- [LLM Pretraining on 500-shard corpus (val)](<https://www.sota2.com/research/sota/llm-pretraining-on-500-shard-corpus-val.md>): LLM Pretraining; 500-shard corpus (val).

- [Pretraining Efficiency on Virtual cell perturbation prediction (train)](<https://www.sota2.com/research/sota/pretraining-efficiency-on-virtual-cell-perturbation-prediction-train.md>): Pretraining Efficiency; Virtual cell perturbation prediction (train).

- [LLM Pretraining on MLPerf C4](<https://www.sota2.com/research/sota/llm-pretraining-on-mlperf-c4.md>): LLM Pretraining; MLPerf C4.