OpenWebText
Benchmarks
TaskDatasetSOTA resultTrendResults
OpenWebText
0.98Iso
18
OpenWebText GPT-2 (val)
16.11Perplexity
17
OpenWebText standard (test)
20.08Perplexity
17
OpenWebText (held-out set)
11.5PPL
16
OpenWebText
0.003Final Loss
15
OpenWebText 1M steps (val)
16.7Validation Perplexity (PPL)
13
OpenWebText GPT-2 (test)
17.94Perplexity
13
OpenWebText (test)
3.77Average Perplexity
13
OpenWebText
27.9Perplexity
12
OpenWebText L=2048 (test)
13.2Gen. PPL
12
OpenWebText2 (val)
17.12Perplexity
12
OpenWebText Negative prompts (test)
0.59Positivity Score
12
OpenWebText (OWT) GPT-2 tokenizer (val)
15.36PPL
12
OpenWebText 1B tokens
26.67Cumulative Orthogonalization Cost (PFLOPs)
9
OpenWebText Regime WR (val)
52Time to Target (min)
9
OpenWebText (test)
14.8GenPPL (Oracle)
9
OpenWebText (val)
2.15Perplexity (PPL)
9
OpenWebText
0.99MAUVE Score
9
OpenWebText (OWT) (held-out)
2NFE
8
OpenWebText
0.014MAUVE Score
8
OpenWebText GPT-2 124M (val)
3.167LCE
8
OpenWebText (val)
14.2OLMo Perplexity
8
OpenWebText Positive prompts (test)
0.6Negativity Score
8
OpenWebText 1 (val)
14.66Validation Perplexity
8
OpenWebText clean accepted set, length-matched
14.4c-PPL
7