FineWeb
Benchmarks
TaskDatasetSOTA resultTrendResults
FineWeb (val)
1.906Validation Loss
259
FineWeb-Edu
8.32PPL
141
FineWeb 10B (val)
19.54Validation Perplexity
56
FineWeb
3,642.88Throughput
36
FineWeb Temporal
0.985nAOPC
28
FineWeb 100B (val)
20.98Validation Perplexity
28
FineWeb 10B
20.69Validation Perplexity
28
FineWeb 10BT (train)
2.597Loss
26
FineWeb-EDU
2.89LM Loss
19
Fineweb 100B
2.6Perplexity (PPL)
19
FineWeb 10B 8K sequence length
3.317Validation Loss
16
FineWeb 10B 4K sequence length
3.229Validation Loss
16
FineWeb 10B 2K sequence length
3.193Validation Loss
16
FineWeb 10B 1K sequence length
3.171Validation Loss
16
FineWeb held-out
0.259∆PPL
16
FineWeb-Edu 10K steps (val)
3.372Validation Loss
15
FineWeb-Edu 10K steps (train)
3.338Training Loss
15
FineWeb-Edu 300M×5B (val)
23.6PPL
15
FineWeb-Edu
71,600Throughput (tokens/s)
15
Fineweb-edu 1.0 (test)
0.002ECE
13
FineWeb (test)
1.93Test Loss
12
FineWeb-Edu (pre-training)
14.13Perplexity
12
FineWeb-Edu
8.49Perplexity (BF16)
12
FineWeb-Edu
1Rank
11
FineWeb Ch Edu
1Rank
11