Deductive logical reasoning on FOLIO 203 (dev)
6.4Exclusion RateICL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ICLModel=Qwen3-4B-Instruct-2507, Inference Type=In-context learning (5-shots)2026.01 | 6.4 | 3.45 | |
| ICLModel=Qwen2.5-3B-Instruct, Inference Type=In-context learning (5-shots)2026.01 | 27.59 | 19.21 | |
| ICLModel=Phi-4-mini-Instruct, Inference Type=In-context learning (5-shots)2026.01 | 45.32 | 28.08 | |
| IncrementModel=Qwen2.5-3B-Instruct, Inference Type=Incremental inference2026.01 | 55.17 | 30.54 | |
| SFT+Model=Qwen2.5-3B-Instruct, Inference Type=SFT on symbolic ProofWriter and FOLIO combined2026.01 | 57.63 | 30.05 | |
| ICLModel=Gemma-3-4B-Instruct, Inference Type=In-context learning (5-shots)2026.01 | 58.13 | 33 | |
| IncrementModel=Qwen3-4B-Instruct-2507, Inference Type=Incremental inference2026.01 | 58.13 | 38.42 | |
| SFT+Model=Qwen3-4B-Instruct-2507, Inference Type=SFT on symbolic ProofWriter and FOLIO combined2026.01 | 60.1 | 40.39 | |
| SFT+Model=Phi-4-mini-Instruct, Inference Type=SFT on symbolic ProofWriter and FOLIO combined2026.01 | 61.08 | 38.92 | |
| IncrementModel=Phi-4-mini-Instruct, Inference Type=Incremental inference2026.01 | 62.07 | 38.92 | |
| SFT+Model=Gemma-3-4B-Instruct, Inference Type=SFT on symbolic ProofWriter and FOLIO combined2026.01 | 66 | 36.95 | |
| IncrementModel=Gemma-3-4B-Instruct, Inference Type=Incremental inference2026.01 | 69.95 | 40.89 |