Text-to-SQL on Spider Lite 2.0 (full)
57.6Overall EXAgentSM
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| AgentSMModel=Claude 3.5 Sonnet, Gold Tables=true, Avg. Steps=15.8, Avg. Input Toks=268K, Avg. Output Toks=5K, Avg Latency (s)=194.32026.01 | 57.6 | 62 | 44 | 79.2 | |
| AgentSMModel=Claude 3.5 Sonnet, Avg. Steps=16.4, Avg. Input Toks=300K, Avg. Output Toks=5K, Avg Latency (s)=247.12026.01 | 44.8 | 52.2 | 35 | 51.9 | |
| AgentSMModel=Claude 3 Opus, Avg. Steps=16.8, Avg. Input Toks=299K, Avg. Output Toks=5K, Avg Latency (s)=226.42026.01 | 38.4 | 40.5 | 33.3 | 42.2 | |
| SpiderAgentModel=Claude 3.5 Sonnet, Avg. Steps=18.9, Avg. Input Toks=200K, Avg. Output Toks=4K, Avg Latency (s)=363.22026.01 | 28.7 | 37.6 | 12.6 | 40.7 | |
| SpiderAgentModel=Claude 3 Opus2026.01 | 27.8 | — | — | — | |
| CodingAgentModel=Claude 3.5 Sonnet, Avg. Steps=18.1, Avg. Input Toks=200K, Avg. Output Toks=4K, Avg Latency (s)=325.52026.01 | 24.7 | 27.3 | 11.1 | 42.2 |