Symbolic Reasoning on MMLU Redux
80.3AccuracyQuaSAR
Evaluation Results
| Method | Links | |
|---|---|---|
| QuaSARBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 80.3 | |
| COTBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 80 | |
| COMATBackbone=Qwen2-7B, Protocol=ICL2025.02 | 79.8 | |
| COTBackbone=Qwen2-7B, Protocol=ICL2025.02 | 79.2 | |
| QuaSARBackbone=Qwen2-7B, Protocol=ICL2025.02 | 75.9 | |
| QuaSARBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 67.2 | |
| Qwen2-7BStrategy=Base, Protocol=Annotation Tuning2025.02 | 66.3 | |
| COTBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 65.9 | |
| Qwen2-7BStrategy=Base, Protocol=ICL2025.02 | 65.5 | |
| COTBackbone=Llama-3-8B, Protocol=ICL2025.02 | 64.5 | |
| Llama-3-8BStrategy=Base, Protocol=Annotation Tuning2025.02 | 63 | |
| QuaSARBackbone=Llama-3-8B, Protocol=ICL2025.02 | 63 | |
| Llama-3-8BStrategy=Base, Protocol=ICL2025.02 | 60.2 | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 49.5 | |
| COTBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 47.8 | |
| COTBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 46.3 | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 44.8 | |
| Qwen2-1.5BStrategy=Base, Protocol=Annotation Tuning2025.02 | 42.4 | |
| Qwen2-1.5BStrategy=Base, Protocol=ICL2025.02 | 41.7 | |
| QuaSARBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 40 | |
| COTBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 34.7 | |
| QuaSARBackbone=Llama-3-1B, Protocol=ICL2025.02 | 34.2 | |
| COTBackbone=Llama-3-1B, Protocol=ICL2025.02 | 34 | |
| Llama-3-1BStrategy=Base, Protocol=Annotation Tuning2025.02 | 30.1 | |
| Llama-3-1BStrategy=Base, Protocol=ICL2025.02 | 28.3 |