Tabular Reasoning on DTR-Bench
1.93Accuracy Win RateDTR (DS-v3)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DTR (DS-v3)Backbone=DS-v3, Category=DTR, Avg. Runtime (s)=62.09, Total Output Tokens=81,754,2, Avg. LLM Calls=4.72026.03 | 1.93 | 37.53 | 30.23 | 30.23 | 27.62 | 27.64 | 42.74 | 42.64 | |
| TreeThinkerCategory=Workflow, Avg. Runtime (s)=140.78, Total Output Tokens=17,067,00, Avg. LLM Calls=5.12026.03 | 1.83 | 31 | 22.82 | 22.82 | 21.42 | 21.43 | 36.83 | 36.83 | |
| Code LoopCategory=Workflow, Avg. Runtime (s)=175.84, Total Output Tokens=75,204,2, Avg. LLM Calls=8.82026.03 | 1.32 | 27.5 | 9.4 | 9.51 | 14.81 | 14.92 | 20.42 | 20.42 | |
| DeepSeek-V3.2Category=Common LLM, Avg. Runtime (s)=39.02, Total Output Tokens=52,446,3, Avg. LLM Calls=1.02026.03 | 1.28 | 33.52 | 25.22 | 25.22 | 24.42 | 24.43 | 36.63 | 36.83 | |
| DeepSeek-V3Category=Common LLM, Avg. Runtime (s)=38.64, Total Output Tokens=49,271,1, Avg. LLM Calls=1.02026.03 | 1.21 | 30.2 | 21.72 | 21.72 | 20.6 | 20.6 | 31.23 | 31.23 | |
| ST-RaptorCategory=Workflow, Avg. Runtime (s)=999.16, Total Output Tokens=31,077,2, Avg. LLM Calls=9.22026.03 | 0.62 | 22.4 | 6 | 6 | 7.41 | 7.41 | 12.4 | 12.4 | |
| TableGPT2-7BCategory=Table specific LLM, Avg. Runtime (s)=3.42, Total Output Tokens=12,450, Avg. LLM Calls=1.02026.03 | 0.2 | 8.41 | 5.12 | 5.12 | 4.33 | 4.35 | 6.21 | 6.21 | |
| TableLLM-7BCategory=Table specific LLM, Avg. Runtime (s)=3.15, Total Output Tokens=10,820, Avg. LLM Calls=1.02026.03 | 0.15 | 6.22 | 3.84 | 3.84 | 3.1 | 3.12 | 4.55 | 4.55 | |
| StructGPTCategory=Table specific LLM, Avg. Runtime (s)=4.88, Total Output Tokens=8,420, Avg. LLM Calls=1.02026.03 | 0.1 | 4.15 | 2.1 | 2.1 | 1.85 | 1.86 | 2.3 | 2.3 |