Spider
Benchmarks
TaskDatasetSOTA resultTrendResults
Spider (test)
91.2Execution Accuracy
256
Spider (dev)
90.3EX
196
Spider
98.65Exec Acc (All)
139
Spider-DK
81.9Execution Accuracy (EX)
136
Spider-Syn
83.1Execution Accuracy (EX)
120
Spider 1.0 (test)
91.2EM Acc (Overall)
110
Spider 1.0 (dev)
86.75Exact Match Accuracy
92
Spider Realistic
88.6Execution Accuracy (EX)
55
Spider to SynSQL 2.5M
2.8MAE
54
Spider to BIRD
3.1MAE
54
Spider-Realistic
88.6Execution Accuracy (EX)
50
Spider Lite 2.0
69.3Execution Accuracy (EX)
46
Spider
41.73Attack Accuracy
40
Spider
88.4Execution Accuracy (EX)
38
Spider (test)
88.3Exec Acc (Greedy)
37
Spider Non-Synthesized Matched Set
97.31Execution Match Accuracy (ExM)
32
Spider
116True Positive Repairs (EM)
30
Spider Non-Seed Domains
64.7Non-Seed Domain Score
30
Spider all domains
58.3SemanticScore
30
Spider
82.5Accuracy
28
Spider 2.0 (test)
26.7Execution Accuracy (Spider 2.0 Test)
27
Spider
63.21AUPRC
24
Spider
1.07AVGLEN
22
Spider 2.0
4.4Llama-8B Execution Accuracy
21
Spider Snow 2.0 (test)
60.51Execution Accuracy (EX)
21