Logical Reasoning on LogiQA (Acc@t1, Acc@t2, Delta)
46.6Acc@t1SFT based
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SFT basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ SFT qa pairs2025.05 | 46.6 | — | — | |
| Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/o reflection2025.05 | 46.4 | — | — | |
| Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/ reflection (variant 1)2025.05 | 46.4 | 56.2 | 9.8 | |
| Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/ reflection (variant 2)2025.05 | 46.4 | 52.8 | 6.4 | |
| Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/o reflection2025.05 | 42.4 | — | — | |
| Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/ reflection (variant 1)2025.05 | 42.4 | 64.4 | 22 | |
| Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/ reflection (variant 2)2025.05 | 42.4 | 56.8 | 14.4 | |
| Prompt basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/o reflection2025.05 | 30.2 | — | — | |
| Prompt basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ reflection2025.05 | 30.2 | 38.8 | 8.6 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=one-stage w/ D+2025.05 | 30.2 | 43.8 | 13.6 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=two-stage w/ D+2025.05 | 30.2 | 49.4 | 19.2 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ D+/-2025.05 | 30.2 | 41.8 | 11.6 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ Dpref2025.05 | 30.2 | 39.2 | 9 | |
| Prompt basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/o reflection2025.05 | 28.8 | — | — | |
| Prompt basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ reflection2025.05 | 28.8 | 31.2 | 2.4 | |
| SFT basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ SFT qa pairs2025.05 | 28.8 | 34.2 | 5.4 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=one-stage w/ D+2025.05 | 28.8 | 38.4 | 9.6 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=two-stage w/ D+2025.05 | 28.8 | 48.8 | 20 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ D+/-2025.05 | 28.8 | 39.2 | 10.4 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ Dpref2025.05 | 28.8 | 38 | 9.2 |