Logical Reasoning on PARARULE-Plus depth-stratified sample 400 instances
100AccuracyReasoning
Evaluation Results
| Method | Links | |
|---|---|---|
| ReasoningBackbone=Claude, Input tokens=717, Output tokens=240, Total tokens=957, Inference time (sec)=4.642026.06 | 100 | |
| Formalizer (Prolog)Backbone=Claude, Input tokens=1003, Output tokens=423, Total tokens=1426, Inference time (sec)=5.112026.06 | 100 | |
| Formalizer (Prolog)Backbone=GPT/o4-mini, Input tokens=881, Output tokens=308, Total tokens=1189, Inference time (sec)=4.232026.06 | 100 | |
| ReasoningBackbone=GPT/o4-mini, Input tokens=652, Output tokens=896, Total tokens=1548, Inference time (sec)=11.272026.06 | 99.8 | |
| Standard LLMBackbone=Claude, Input tokens=688, Output tokens=160, Total tokens=849, Inference time (sec)=3.572026.06 | 98.8 | |
| Standard LLMBackbone=GPT/o4-mini, Input tokens=653, Output tokens=2, Total tokens=655, Inference time (sec)=0.982026.06 | 76.2 |