Fineweb-edu
Benchmarks
TaskDatasetSOTA resultTrendResults
FineWeb-Edu (test)
16.46Perplexity (Test)
58
FineWeb-Edu (val)
8.71Perplexity
51
FineWeb-Edu 500M-token (val)
2.221Valid Loss
18
FineWeb-Edu Mask source 170M-parameter (train)
8.5Entropy
17
FineWeb-Edu Uniform source 170M-parameter (train)
7Entropy
17
FineWeb-EDU (train)
2.993Loss
16
FineWeb-Edu 100B (val)
2.62CE Loss
13
FineWeb-Edu English, 1.4T tokens (test)
100Similarity Score
12
FineWeb-Edu 100B (eval)
13.75Perplexity
9
Fineweb-Edu 16.8B tokens
36.86ARC-c Accuracy
8
FineWeb-Edu T=4096
41.45Perplexity
7
FineWeb-Edu T=2048
41.65Perplexity
7
FineWeb-Edu T=1024
33.37Perplexity
7
Fineweb-edu distillation 8B to 300M
2.74LM Loss
7
FineWeb-Edu 10B (held-out last unused bin)
15.34Perplexity
5
FineWeb-Edu 10B (val)
3.01Validation CE
5
FineWeb-Edu 20B tokens (val)
15.03Final PPL
3
FineWeb-edu 12 text passages (held-out)
1.683Average Loss
3
FineWeb-Edu 1.4B tokens (val)
3.271Loss
3