OpenWebText
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
OpenWebText full 1000-sample pool
0Repetition Percentage
7
OpenWebText GPT-2 125M (train)
0.25Peak Memory Footprint
7
OpenWebText 30k (val)
2.934Loss (nats/token)
6
OpenWebText
230.673L0
6
OpenWebText (OWT) length 1024
2.9Training Duration (days)
6
OpenWebText (held-out segments)
1MAUVE
6
OpenWebText
1MAUVE
6
OpenWebText (test val)
19.6Perplexity
6
OpenWebText 1B tokens Gemma-2-2B (train)
0.963KL Score
5
OpenWebText (val)
3.93NLL (Context=1024)
5
OpenWebText
0.616MAUVE
5
OpenWebText 100k steps (val)
3.004Validation Loss (100k steps)
5
OpenWebText (test)
-0.07Tail-patch Score
5
OpenWebText (val)
18.8Perplexity
5
OpenWebText (train)
18.4PPL
5
OpenWebText (dev)
18.32Perplexity (Dev)
4
OpenWebText padded (test)
3.74NLL
4
OpenWebText
59.9MAUVE
4
OpenWebText
0p-value
4
OpenWebText (train)
0Logits Output (t-test) P-value
4
OpenWebText Positive Prompts (test)
3.32Generation Score
4
OpenWebText
3.41Clarity
4
OpenWebText ctx len. 1024 (val)
21.43Perplexity
4
OpenWebText GPT-2 124M NanoGPT (train)
2.955Loss
4
OpenWebText
7.61BPD
4