Symbolic Reasoning on OlyBench
36.5AccuracyQuaSAR
Evaluation Results
| Method | Links | |
|---|---|---|
| QuaSARBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 36.5 | |
| COMATBackbone=Qwen2-7B, Protocol=ICL2025.02 | 32.2 | |
| QuaSARBackbone=Qwen2-7B, Protocol=ICL2025.02 | 27.8 | |
| QuaSARBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 16.6 | |
| COTBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 14.7 | |
| QuaSARBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 14.6 | |
| Llama-3-8BStrategy=Base, Protocol=Annotation Tuning2025.02 | 13.2 | |
| QuaSARBackbone=Llama-3-8B, Protocol=ICL2025.02 | 13 | |
| COTBackbone=Llama-3-8B, Protocol=ICL2025.02 | 12.4 | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 11.8 | |
| Llama-3-8BStrategy=Base, Protocol=ICL2025.02 | 10.9 | |
| Qwen2-7BStrategy=Base, Protocol=Annotation Tuning2025.02 | 10.9 | |
| COTBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 10.7 | |
| COTBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 10.6 | |
| Qwen2-7BStrategy=Base, Protocol=ICL2025.02 | 10.5 | |
| QuaSARBackbone=Llama-3-1B, Protocol=ICL2025.02 | 9.8 | |
| COTBackbone=Qwen2-7B, Protocol=ICL2025.02 | 9.8 | |
| COTBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 9.1 | |
| COTBackbone=Llama-3-1B, Protocol=ICL2025.02 | 8.2 | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 8.2 | |
| COTBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 7.8 | |
| Qwen2-1.5BStrategy=Base, Protocol=Annotation Tuning2025.02 | 7.4 | |
| Llama-3-1BStrategy=Base, Protocol=Annotation Tuning2025.02 | 7.1 | |
| Qwen2-1.5BStrategy=Base, Protocol=ICL2025.02 | 6.9 | |
| Llama-3-1BStrategy=Base, Protocol=ICL2025.02 | 6.5 |