Pythia
Benchmarks
TaskDatasetSOTA resultTrendResults
Pythia 1B (16L), 410M (24L) (val)
4.4ΔPPL
36
Pythia model activations
0.807Pearson Correlation Coefficient
18
Pythia 6.9B
13.1TPR@1%FPR
16
Pythia-160M
14.47Delta Perplexity (ΔPPL)
16
Pythia-70M layer 3
0.342Relative Error
13
Pythia 6.9B
89.31Memory Usage (%)
9
Pythia 2.8B
5.94Memory Usage (%)
9
Pythia ≤6.9B (k=5)
0.847Pooled R^2
8
Pythia k=4 (≤2.8B)
0.837Pooled R^2
8
Pythia k=3 (1B, 410M, 160M)
0.605Pooled R^2
8
Pythia quanto 2-bit
0.903R2 Score
8
Pythia bnb 4-bit
99.36R2 Score
8
Pythia AWQ 4-bit
0.994R2 Score
8
Pythia Predict 272.6B–307B
0.945Pooled R2
8
Pythia Predict 180.4B–307B
0.781Pooled R2
8
Pythia Predict 75.5B–307B
80.5Pooled R2
8
Pythia Suite
99.53Performance (4-bit)
7
Pythia-160m layer 8 residual-stream activations
9.7Delta CE
6
Pythia-1B activations
0.017Delta R^2
6
Pythia-12B
54.82Benign F1 Score
5
Pythia-70M Layer 3 activations (held-out)
0.342Relative Error
4
Pythia activations (The Pile) 160M (layer 12)
0.148MSE
4
Pythia attention output projections 1.4B
0.269Dimensionless Reconstruction Error Fraction
4
Pythia attention output projections 70M
16.8Reconstruction Error Fraction
4
Pythia-1.4b
76.2Avg. F1
4