Spider
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
Spider T1w (test)
82Global Dice Score
9
Spider SmBoP parser outputs (dev)
78EM
9
Spider BRIDGEv2 parser outputs (dev)
72.5EM
9
Spider CodeT5 parser outputs (dev)
69.2EM
9
Spider-Realistic 1.0 (test)
77.4Exact Match (EM)
9
Spider (test)
12SELECT Component Error
9
Spider
53.6Accuracy
8
SPIDER
28.31PSNR
8
Spider2 Snow
69.29Success Rate
8
Spider2-Snow (test)
97.88SRR
8
Spider2-Snow
90.15SRR
8
Spider2-Lite SQLite
62.59Pass@1 Accuracy
8
Spider 1 (test)
65.7Precision@5
8
Spider 1 (dev)
85.2PR@5
8
Spider
39.5T-EM
7
Spider (test)
84mIoU
7
Spider schema presentation perturbations (dev)
95.7AST Similarity (Schema)
7
Spider 200 question subset with paraphrase perturbations (dev)
0.894AST Similarity (para)
7
Spider 600 Q (test dev)
73.83Execution Accuracy
7
Spider
3Adjusted bits/Byte
7
Spider-Syn 1.0 (test)
69.1Exact Match
7
Spider-Realistic 1.0 (val)
72.4EM
7
Spider
55Original Pipeline Repairs
6
Spider2-Snow 2.0 (test)
55.15Pass@1
6
Spider 1 (train)
55Precision@5
6