Question Answering on TabMWP (test)
98.78Accuracy (All)Chameleon (GPT-4)
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Chameleon (GPT-4)Protocol=Few-shot, Tuned Params=0M, Backbone=GPT-42023.04 | 98.78 | 98.95 | 98.29 | 99.34 | 97.42 | 98.58 | 98.56 | 93.33 | 98.95 | 98.54 | |
| GPT-4 PoTProtocol=Few-shot, Tuned Params=0M2023.04 | 96.93 | 97.4 | 95.58 | 98.48 | 93.22 | 96.25 | 98 | 68.57 | 96.97 | 96.87 | |
| Chameleon (ChatGPT)Protocol=Few-shot, Tuned Params=0M, Backbone=ChatGPT2023.04 | 93.28 | 93.13 | 93.72 | 92.71 | 94.76 | 91.29 | 98.11 | 78.85 | 93.37 | 93.17 | |
| GPT-4 CoTProtocol=Few-shot, Tuned Params=0M2023.04 | 90.81 | 88.48 | 97.49 | 86.16 | 97.51 | 96.86 | 99.11 | 89.52 | 92.4 | 88.7 | |
| Human performanceProtocol=Heuristic baselines2023.04 | 90.22 | 84.61 | 93.32 | 84.95 | 83.29 | 97.18 | 88.69 | 96.2 | 94.27 | 81.28 | |
| ChatGPT PoTProtocol=Few-shot, Tuned Params=0M2023.04 | 89.49 | 90.24 | 87.35 | 89.31 | 93.82 | 92.1 | 85.89 | 55.24 | 90.6 | 88 | |
| ChatGPT CoTProtocol=Few-shot, Tuned Params=0M2023.04 | 82.03 | 78.43 | 92.32 | 75.38 | 90.3 | 92.3 | 92.89 | 87.62 | 83.06 | 80.66 | |
| Codex PoT-SCProtocol=Few-shot, Tuned Params=0M, subset=true2023.04 | 81.8 | — | — | — | — | — | — | — | — | — | |
| Codex PoTProtocol=Few-shot, Tuned Params=0M, subset=true2023.04 | 73.2 | — | — | — | — | — | — | — | — | — | |
| GPT-3 CoT-PromptPGProtocol=Few-shot, Tuned Params=0M2023.04 | 68.23 | 66.17 | 74.11 | 64.12 | 74.16 | 76.19 | 72.81 | 65.71 | 71.2 | 64.27 | |
| GPT-3 CoTProtocol=Few-shot, Tuned Params=0M2023.04 | 62.92 | 60.76 | 69.09 | 60.04 | 63.58 | 76.49 | 61.19 | 67.3 | 68.62 | 55.31 | |
| CodexProtocol=Few-shot, Tuned Params=0M, subset=true2023.04 | 59.4 | — | — | — | — | — | — | — | — | — | |
| TAPEX-LARGEProtocol=Fine-tuned, Tuned Params=406M2023.04 | 58.52 | 51 | 80.02 | 59.92 | 16.31 | 95.34 | 64 | 73.33 | 67.11 | 47.07 | |
| GPT-3 CoTProtocol=Zero-shot, Tuned Params=0M2023.04 | 57.61 | 54.36 | 66.92 | 55.82 | 48.67 | 78.82 | 55.67 | 51.43 | 63.62 | 49.59 | |
| UnifiedQA-LARGEProtocol=Fine-tuned, Tuned Params=738M2023.04 | 57.35 | 48.67 | 82.18 | 55.97 | 20.26 | 94.63 | 68.89 | 79.05 | 65.92 | 45.92 | |
| GPT-3Protocol=Few-shot, Tuned Params=0M2023.04 | 57.13 | 54.69 | 64.11 | 58.36 | 40.4 | 75.95 | 52.41 | 53.02 | 63.1 | 49.16 | |
| GPT-3Protocol=Zero-shot, Tuned Params=0M2023.04 | 56.96 | 53.57 | 66.67 | 55.55 | 45.84 | 78.22 | 55.44 | 54.29 | 63.37 | 48.41 | |
| TAPEX-BASEProtocol=Fine-tuned, Tuned Params=139M2023.04 | 48.27 | 39.59 | 73.09 | 46.85 | 11.33 | 84.19 | 61.33 | 69.52 | 56.7 | 37.02 | |
| UnifiedQA-BASEProtocol=Fine-tuned, Tuned Params=223M2023.04 | 43.52 | 34.02 | 70.68 | 40.74 | 7.9 | 84.09 | 55.67 | 73.33 | 53.31 | 30.46 | |
| UnifiedQA-SMALLProtocol=Fine-tuned, Tuned Params=41M2023.04 | 29.79 | 22.27 | 51.31 | 27.27 | 2.83 | 52.28 | 48.11 | 69.52 | 35.85 | 21.71 | |
| Heuristic guessProtocol=Heuristic baselines2023.04 | 15.29 | 6.71 | 39.81 | 8.37 | 0.26 | 30.8 | 51.22 | 26.67 | 17.55 | 12.27 |