Natural Language Inference on HANS evaluated subset
157Rule-bearingMechaRule (Qwen2-1.5B)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| MechaRule (Qwen2-1.5B)Condition=exhaustive singletons2026.05 | 157 | 0.911 | 114 | 32.2 | 14.2 | |
| MechaRule (Qwen2-7B)Condition=spectral only2026.05 | 33 | 0.924 | 24 | 61.2 | 19 | |
| MechaRule (Qwen2-1.5B)Condition=spectral only2026.05 | 29 | 0.945 | 25 | 32.2 | 13.2 | |
| MechaRule (Qwen2-7B)Condition=rule + spectral2026.05 | 29 | 0.925 | 21 | 61.5 | 18.6 | |
| MechaRule (Qwen2-7B)Condition=rule + random2026.05 | 28 | 0.915 | 21 | 61.2 | 18.8 | |
| MechaRule (Qwen2-1.5B)Condition=rule + spectral2026.05 | 26 | 0.934 | 24 | 32.2 | 13.4 | |
| MechaRule (Qwen2-7B)Condition=fake rule + spectral2026.05 | 26 | 0.899 | 17 | 61.5 | 17.6 | |
| MechaRule (Qwen2-1.5B)Condition=rule + random2026.05 | 24 | 0.945 | 22 | 32.2 | 13.5 | |
| MechaRule (Qwen2-1.5B)Condition=fake rule + spectral2026.05 | 16 | 0.93 | 14 | 32.2 | 11.5 |