Symbolic Reasoning on Last Letter Concatenation (LLC) out-of-distribution (test)
98LLC-4 AccuracyStrategyLLM
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| StrategyLLMPrompting strategy=Strategy-based2023.11 | 98 | 86.5 | 51.5 | 78.7 | |
| StrategyLLM-SCPrompting strategy=Strategy-based, Sampling strategy=Self-Consistency2023.11 | 98 | 87 | 52.5 | 79.2 | |
| StrategyLLM-ZSPrompting strategy=Strategy-based, Evaluation protocol=Zero-shot2023.11 | 98 | 86 | 44 | 76 | |
| SolutionLLMMethod category=Baseline2023.11 | 81.5 | 25.5 | 0 | 35.7 | |
| CoTPrompting strategy=Chain-of-Thought2023.11 | 68.5 | 22 | 0 | 30.2 | |
| CoT-SCPrompting strategy=Chain-of-Thought, Sampling strategy=Self-Consistency2023.11 | 68 | 22 | 0 | 30 | |
| SPPrompting strategy=Standard2023.11 | 0 | 0 | 0 | 0 |