# ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotation

Source: [ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotation](<https://www.sota2.com/research/paper/6a3a1326206542251d7c70da>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-24T05:00:13.022Z

Authors: Pei-Cing Huang, Chienyu Liu, Chan Hsu, Ci-Siang Chen, Pei-Ju Lee, Yihuang Kang

Published: 2026-06-20T04:10:02.000Z

## Abstract

Current evaluations of Large Language Models (LLMs) on logical fallacy detection focus on predicted labels, but do not establish whether those labels are supported by the reasoning the models provide. We propose ForEx (Formal Verification for Explainable Reasoning), a framework that translates LLM-generated explanations into Lean4 and verifies whether the translated rationale is derivable under encoded premises, not the logical validity of the original natural language argument. To distinguish prediction outcomes from the formal status of the supporting reasoning, we introduce the LLM Argument Verification Matrix, which separates label consistency from formal verification status. Experiments on LOGIC-Climate show that over 90% of LLM outputs can be translated into formal reasoning chains that pass verification, while agreement with human annotations remains around 20%. These results expose a systematic gap between formal derivability and label agreement, a distinction invisible to prediction-based metrics. ForEx moves LLM evaluation beyond label correctness toward machine-checkable analysis of formalized reasoning chains.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Logical Fallacy Detection on LOGIC-Climate 10% representative subset (107 instances)](<https://www.sota2.com/research/sota/logical-fallacy-detection-on-logic-climate-10-percent-representative-subset-107-instances.md>): Logical Fallacy Detection; LOGIC-Climate 10% representative subset (107 instances); Count: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.21867>)

[PDF](<https://arxiv.org/pdf/2606.21867.pdf>)
