Copy
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
Copy OOD lengths: 2x, 4x, 8x, 16x, 32x, 64x
100Exact Match Accuracy
30
Copy Length 120
100Accuracy
7
Copy d=2000
100Accuracy
6
Copy d=500
100Accuracy
6
Copy Eval
94.7Hr
6
Copy Length 5020
99.3Accuracy
6
Copy ID, n=64
100Exact Match Accuracy
5
Copy (OOD lengths 20-60)
68.7OOD Score
4
Copy (test)
1Perplexity (PPL)
4
Copy
1PPL
4
Copy (test)
100Per Character Accuracy
4