Reasoning Evaluation Suite Math, Symbolic, and Commonsense (test)
80.8Math AccuracyDerailer + Rerailer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Derailer + RerailerModel=Claude-3.5-Sonnet2024.08 | 80.8 | 89.7 | 75.9 | 82.1 | 592 | 15.88 | |
| Chain of Ver (CoVe)Model=Claude-3.5-Sonnet2024.08 | 78.2 | 86.8 | 75.4 | 80.1 | 1,778 | 4.16 | |
| Tree of Thought (ToT)Model=Claude-3.5-Sonnet2024.08 | 78.1 | 86.7 | 75.4 | 80.1 | 2,245 | 3.3 | |
| RerailerModel=Claude-3.5-Sonnet2024.08 | 78 | 86.7 | 75.3 | 80 | 1,458 | 5.01 | |
| Derailer + CoVeModel=Claude-3.5-Sonnet2024.08 | 77.9 | 86.6 | 75.3 | 79.9 | 793 | 9.08 | |
| Self-Consistency (SC)Model=Claude-3.5-Sonnet2024.08 | 77.8 | 86.5 | 75.2 | 79.8 | 1,732 | 4.1 | |
| Derailer + ToTModel=Claude-3.5-Sonnet2024.08 | 77.8 | 86.5 | 75.2 | 79.8 | 845 | 8.4 | |
| Derailer + SCModel=Claude-3.5-Sonnet2024.08 | 77.6 | 86.3 | 75.1 | 79.7 | 762 | 9.19 | |
| Five-shot CoTModel=Claude-3.5-Sonnet2024.08 | 72.9 | 81.4 | 74.8 | 76.4 | 518 | 7.14 | |
| Derailer + RerailerModel=GPT-4o-mini2024.08 | 72 | 58.5 | 73.8 | 68.1 | 639 | 15.02 | |
| Least-to-Most CoTModel=Claude-3.5-Sonnet2024.08 | 70.4 | 79.3 | 74.1 | 74.6 | 372 | 5.11 | |
| Chain of Ver (CoVe)Model=GPT-4o-mini2024.08 | 69.3 | 55.5 | 73.2 | 66 | 1,842 | 4.07 | |
| Tree of Thought (ToT)Model=GPT-4o-mini2024.08 | 69.2 | 55.4 | 73.1 | 65.9 | 2,325 | 3.18 | |
| RerailerModel=GPT-4o-mini2024.08 | 69 | 55.4 | 73.1 | 65.8 | 1,539 | 4.74 | |
| Derailer + CoVeModel=GPT-4o-mini2024.08 | 68.9 | 55.3 | 73.1 | 65.8 | 864 | 8.45 | |
| Derailer + ToTModel=GPT-4o-mini2024.08 | 68.8 | 55.2 | 73 | 65.7 | 912 | 7.89 | |
| Self-Consistency (SC)Model=GPT-4o-mini2024.08 | 68.6 | 55.2 | 72.9 | 65.6 | 1,795 | 3.96 | |
| Zero-shot CoTModel=Claude-3.5-Sonnet2024.08 | 68.3 | 77.6 | 72.2 | 72.7 | 108 | — | |
| Derailer + SCModel=GPT-4o-mini2024.08 | 68.3 | 54.9 | 72.8 | 65.3 | 823 | 8.26 | |
| Five-shot CoTModel=GPT-4o-mini2024.08 | 63.9 | 50.8 | 72.4 | 62.4 | 556 | 7.01 | |
| Least-to-Most CoTModel=GPT-4o-mini2024.08 | 62.4 | 49.1 | 71.8 | 61.1 | 389 | 6.68 | |
| Zero-shot CoTModel=GPT-4o-mini2024.08 | 59.5 | 46.8 | 69.2 | 58.5 | 121 | — | |
| Derailer + RerailerModel=Llama-3.1 70B2024.08 | 48.4 | 81.8 | 74.2 | 68.1 | 648 | 12.5 | |
| Chain of Ver (CoVe)Model=Llama-3.1 70B2024.08 | 46.2 | 79.4 | 73.6 | 66.4 | 1,972 | 3.25 | |
| Tree of Thought (ToT)Model=Llama-3.1 70B2024.08 | 46.1 | 79.3 | 73.5 | 66.3 | 2,458 | 2.56 | |
| RerailerModel=Llama-3.1 70B2024.08 | 46.1 | 79.3 | 73.5 | 66.3 | 1,652 | 3.81 | |
| Derailer + CoVeModel=Llama-3.1 70B2024.08 | 46 | 79.2 | 73.5 | 66.2 | 923 | 6.72 | |
| Self-Consistency (SC)Model=Llama-3.1 70B2024.08 | 45.9 | 79.1 | 73.4 | 66.1 | 1,932 | 3.16 | |
| Derailer + ToTModel=Llama-3.1 70B2024.08 | 45.9 | 79.1 | 73.4 | 66.1 | 968 | 6.3 | |
| Derailer + SCModel=Llama-3.1 70B2024.08 | 45.7 | 78.8 | 73.3 | 65.9 | 892 | 6.61 | |
| Five-shot CoTModel=Llama-3.1 70B2024.08 | 41.8 | 74.7 | 72.9 | 63.1 | 628 | 4.94 | |
| Least-to-Most CoTModel=Llama-3.1 70B2024.08 | 39.6 | 72.4 | 72.3 | 61.4 | 448 | 3.13 | |
| Zero-shot CoTModel=Llama-3.1 70B2024.08 | 38.2 | 71.3 | 70.4 | 60 | 132 | — |