List your company
ResearchFormerly WizWand

Logical Reasoning on LogiQA (Acc@t1, Acc@t2, Delta)

46.6Acc@t1SFT based
28.08832.89437.742.506May 22, 2025
Updated 8mo ago

Evaluation results

MethodLinks
SFT basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ SFT qa pairs2025.0546.6——
Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/o reflection2025.0546.4——
Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/ reflection (variant 1)2025.0546.456.29.8
Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/ reflection (variant 2)2025.0546.452.86.4
Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/o reflection2025.0542.4——
Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/ reflection (variant 1)2025.0542.464.422
Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/ reflection (variant 2)2025.0542.456.814.4
Prompt basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/o reflection2025.0530.2——
Prompt basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ reflection2025.0530.238.88.6
ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=one-stage w/ D+2025.0530.243.813.6
ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=two-stage w/ D+2025.0530.249.419.2
ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ D+/-2025.0530.241.811.6
ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ Dpref2025.0530.239.29
Prompt basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/o reflection2025.0528.8——
Prompt basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ reflection2025.0528.831.22.4
SFT basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ SFT qa pairs2025.0528.834.25.4
ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=one-stage w/ D+2025.0528.838.49.6
ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=two-stage w/ D+2025.0528.848.820
ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ D+/-2025.0528.839.210.4
ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ Dpref2025.0528.8389.2