Deontic Reasoning on DeonticBench SARA-Bin
90.6Macro F1Qwen3.5-397B
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3.5-397BHarness=terminus-kira2026.06 | 90.6 | |
| Qwen3.5-397BHarness=claude-code2026.06 | 88.9 | |
| Kimi-K2Harness=terminus-kira2026.06 | 88.5 | |
| Qwen3.5-35BHarness=terminus-22026.06 | 83.3 | |
| Qwen3.5-35BHarness=claude-code2026.06 | 83.3 | |
| Qwen3.5-397BHarness=terminus-22026.06 | 83.3 | |
| Qwen3.5-35BHarness=terminus-kira2026.06 | 82.9 | |
| Qwen3.5-122BHarness=terminus-kira2026.06 | 82.3 | |
| Qwen3-Coder-480BHarness=claude-code2026.06 | 80 | |
| Qwen3.5-122BHarness=terminus-22026.06 | 80 | |
| Qwen3.5-122BHarness=codex2026.06 | 79.9 | |
| Qwen3-Coder-480BHarness=terminus-22026.06 | 79.3 | |
| Qwen3.5-122BHarness=claude-code2026.06 | 79.3 | |
| Qwen3.5-397BHarness=direct2026.06 | 78.2 | |
| Qwen3-Coder-480BHarness=terminus-kira2026.06 | 76.6 | |
| Qwen3.5-122BHarness=direct2026.06 | 75.3 | |
| Qwen3.5-35BHarness=direct2026.06 | 74 | |
| Kimi-K2Harness=codex2026.06 | 73.3 | |
| Qwen3-235BHarness=codex2026.06 | 72.1 | |
| Kimi-K2Harness=direct2026.06 | 68.4 | |
| Qwen3-235BHarness=direct2026.06 | 66.5 | |
| Qwen3-235BHarness=terminus-kira2026.06 | 66.5 | |
| Qwen3-Coder-480BHarness=codex2026.06 | 59.8 | |
| Qwen3-235BHarness=terminus-22026.06 | 59.8 | |
| GPT-5.2Harness=direct2026.06 | 59.7 | |
| Kimi-K2Harness=terminus-22026.06 | 59.3 | |
| Qwen3-Coder-480BHarness=direct2026.06 | 59.1 | |
| GPT-5.2Harness=terminus-kira2026.06 | 56.9 | |
| GPT-5.2Harness=terminus-22026.06 | 53.1 | |
| GPT-5.2Harness=codex2026.06 | 46.4 |