Symbolic Reasoning on AQUA
80.3AccuracyCOT
Evaluation results
| Method | Links | |
|---|---|---|
| COTBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 80.3 | |
| COTBackbone=Qwen2-7B, Protocol=ICL2025.02 | 79.1 | |
| QuaSARBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 78.4 | |
| QuaSARBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 78.3 | |
| QuaSARBackbone=Qwen2-7B, Protocol=ICL2025.02 | 72.6 | |
| COMATBackbone=Qwen2-7B, Protocol=ICL2025.02 | 72.4 | |
| COTBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 72.2 | |
| COTBackbone=Llama-3-8B, Protocol=ICL2025.02 | 69.6 | |
| Llama-3-8BStrategy=Base, Protocol=Annotation Tuning2025.02 | 67.3 | |
| QuaSARBackbone=Llama-3-8B, Protocol=ICL2025.02 | 67.2 | |
| Llama-3-8BStrategy=Base, Protocol=ICL2025.02 | 65.2 | |
| Qwen2-7BStrategy=Base, Protocol=Annotation Tuning2025.02 | 63.7 | |
| FLAREBackbone=Llama-3-8B, Protocol=ICL2025.02 | 62.9 | |
| Qwen2-7BStrategy=Base, Protocol=ICL2025.02 | 62.9 | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 62.2 | |
| COTBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 59.9 | |
| COTBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 58.7 | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 57.6 | |
| Qwen2-1.5BStrategy=Base, Protocol=Annotation Tuning2025.02 | 57.2 | |
| Qwen2-1.5BStrategy=Base, Protocol=ICL2025.02 | 56.8 | |
| QuaSARBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 55.4 | |
| COTBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 52 | |
| QuaSARBackbone=Llama-3-1B, Protocol=ICL2025.02 | 51.6 | |
| COTBackbone=Llama-3-1B, Protocol=ICL2025.02 | 50.7 | |
| Llama-3-1BStrategy=Base, Protocol=Annotation Tuning2025.02 | 40.3 | |
| Llama-3-1BStrategy=Base, Protocol=ICL2025.02 | 39.2 |