Pre-training corpus
Benchmarks
TaskDatasetSOTA resultTrendResults
Pre-training corpus (train)
15.71Perplexity
20
Pre-training corpus
1.577Loss
9
Pre-training corpus (train)
66.4Token Accuracy
9
1.3B 26B-token pre-training corpus (val)
2.077Validation Cross-Entropy
3