PG-19
Benchmarks
TaskDatasetSOTA resultTrendResults
PG-19 (test)
9.765Perplexity
112
PG-19
407.96Throughput
64
PG-19 24K context length
393.9Throughput
32
PG-19 16K context length
486.5Throughput
32
PG-19 8K context length
553.79Throughput
32
PG-19 mini 10K context
100Accuracy (Needle-in-the-Haystack)
30
PG-19 (val)
18.43Perplexity
29
PG-19 500M parameters scale (test)
40.72PPLX
20
PG-19 (Whole Book)
18.87PPL @ 50K
17
PG-19 mini 100K context
100Accuracy
15
PG-19 mini 32K context
100Accuracy
15
PG-19 subword-level
3.94Forward BPT
6
PG-19 60K context length
6.29Throughput Speedup (micro)
6
PG-19 50K context length
5.79Throughput Speedup (micro)
6
PG-19 40K context length
5.46Throughput Speedup (micro)
6
PG-19 30K context length
4.75Throughput Speedup (micro)
6
PG-19 (dev)
52.08Perplexity
6
PG-19 (test)
1,568Max Tokens
6
PG-19 long-context
101.09Perplexity (PPL)
5
PG-19 128K context length
7.244Perplexity
2
PG-19 64K context length
9.043Perplexity
2
PG-19 8K context length
12.313Perplexity
2