Deontic Reasoning on DeonticBench USCIS-AAO
81.9Macro F1GPT-5.2
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-5.2Harness=codex2026.06 | 81.9 | |
| GPT-5.2Harness=terminus-22026.06 | 78.1 | |
| Qwen3.5-122BHarness=direct2026.06 | 78 | |
| GPT-5.2Harness=direct2026.06 | 77.9 | |
| Qwen3.5-397BHarness=terminus-kira2026.06 | 77.8 | |
| Qwen3.5-122BHarness=codex2026.06 | 77.5 | |
| Qwen3.5-122BHarness=terminus-kira2026.06 | 76.4 | |
| Qwen3.5-122BHarness=claude-code2026.06 | 73 | |
| Qwen3.5-397BHarness=direct2026.06 | 72.7 | |
| Qwen3.5-35BHarness=terminus-kira2026.06 | 71.8 | |
| GPT-5.2Harness=terminus-kira2026.06 | 71.3 | |
| Qwen3.5-397BHarness=terminus-22026.06 | 70.8 | |
| Qwen3-235BHarness=terminus-22026.06 | 68.9 | |
| Qwen3-235BHarness=terminus-kira2026.06 | 66.8 | |
| Kimi-K2Harness=terminus-kira2026.06 | 66.8 | |
| Qwen3.5-397BHarness=claude-code2026.06 | 64.3 | |
| Qwen3.5-35BHarness=terminus-22026.06 | 60.7 | |
| Qwen3.5-122BHarness=terminus-22026.06 | 60.3 | |
| Qwen3.5-35BHarness=claude-code2026.06 | 60.3 | |
| Kimi-K2Harness=codex2026.06 | 55.3 | |
| Kimi-K2Harness=terminus-22026.06 | 53.3 | |
| Qwen3-235BHarness=direct2026.06 | 53.1 | |
| Kimi-K2Harness=direct2026.06 | 51.8 | |
| Qwen3-235BHarness=codex2026.06 | 50.9 | |
| Qwen3-Coder-480BHarness=claude-code2026.06 | 50.5 | |
| Qwen3.5-35BHarness=direct2026.06 | 47.7 | |
| Qwen3-Coder-480BHarness=codex2026.06 | 42.7 | |
| Qwen3-Coder-480BHarness=terminus-22026.06 | 40.8 | |
| Qwen3-Coder-480BHarness=terminus-kira2026.06 | 37.8 | |
| Qwen3-Coder-480BHarness=direct2026.06 | 33.8 |