Spider
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
Spider
50.4Execution Accuracy
6
Spider
60.5Execution Accuracy
6
Spider SQLite 2.0 (dev)
15.6Greedy Execution Accuracy
6
Spider
74.4Pass@1
6
Spider Snow header-masking 2.0
86Execution Accuracy
6
Spider Snow (style-shift) 2.0
86Execution Accuracy
6
Spider Snow 2.0
0.78EXAcc
6
Spider Lite 2.0 (full)
57.6Overall EX
6
Spider lite 2.0 (first 100 queries)
52.4Easy Execution Accuracy
6
Spider-Syn (test)
72QM (%)
6
Spider Robustness Suite SYN REALISTIC DK (dev)
78.9Execution Accuracy (SYN)
6
Spider
76.4Pass@1
5
Spider Snow 2.0
65.44Majority@K
5
Spider 2.0
9MAE (ATHENA)
5
Spider
8.3MAE (ATHENA)
5
Spider-DK 1.0 (test)
53.3EM
5
Spider2-DBT
37.5Database Match Accuracy
4
Spider Snow 2
69.5Execution Accuracy
4
Spider2-Lite
71.3Execution Accuracy
4
Spider2-Lite Snowflake v1.0 (test)
52.23Pass@1
4
Spider Robustness Variants (SYN, REALISTIC, DK) (test)
76.9Accuracy (SYN)
4
Spider Lite 2.0 (test)
46.6Execution Accuracy (EX)
4
Spider SQL 2.0
72Execution Success Rate
4
Spider lite 2.0 (Random 100 examples)
28Execution Accuracy (EX)
4
Spider 120 2018/2025 (test)
88.3SRR
4