Categories Companies Research

Loading the SOTA2 catalog…

Explore

Categories Companies Research

Contribute

Join SOTA2 Contact

SOTA2

© 2026 SOTA2

Build in Bellevue, USA

WizWand is now SOTA2 Research

Papers Benchmarks Datasets Tasks

Language Model Pre-training research benchmarks · SOTA2 Research

Language Model Pre-training

Benchmarks

Dataset NameSOTA MethodSort

Dataset NameSOTA methodMetricTrendResultsLast Updated

C4 Llama 2 pre-training (val)

13.19Perplexity

47

Feb 26, 2026

C4 Llama-160M scratch (val)

3.091Validation Loss

20

Feb 26, 2026

C4 Llama-1B scratch (val)

2.638Validation Loss

16

Feb 26, 2026

3.86Eval Loss

12

Jun 30, 2026

FineWeb-Edu (pre-training)

14.13Perplexity

12

May 12, 2026

BERT-Large NVIDIA V100 (train)

96Max Batch Size

6

Feb 26, 2026

BERT-Large NVIDIA 2080 Ti (train)

50Max Batch Size

6

Feb 26, 2026

Datacomp-LM 400M-1x scale 1.0

DCLM-baseline + SIEVE

17.9Core Score

3

Feb 26, 2026

Popular datasets

C4 BERT FineWeb Datacomp-LM

Follow for updates

Join SOTA2 Contact