# Deontic Reasoning on DeonticBench USCIS-AAO Source: [Deontic Reasoning on DeonticBench USCIS-AAO]() Collection index: [llms.txt]() Data updated: 2026-06-04T13:12:01.690Z Evaluation of model performance on Deontic Reasoning tasks using the DeonticBench USCIS-AAO dataset, measured by Macro F1. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Deontic Reasoning | | Dataset (including any stated split) | DeonticBench USCIS-AAO | | Metric | Macro F1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 30 of 30 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Macro F1](): Higher is better; unit: PERCENT. ## Recorded results | Method | Macro F1 | Result date | Report paper | | --- | --- | --- | --- | | GPT-5.2 | 81.9 | Not recorded | 6a2106eac10484bac23f0605 | | GPT-5.2 | 78.1 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-122B | 78 | Not recorded | 6a2106eac10484bac23f0605 | | GPT-5.2 | 77.9 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-397B | 77.8 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-122B | 77.5 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-122B | 76.4 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-122B | 73 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-397B | 72.7 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-35B | 71.8 | Not recorded | 6a2106eac10484bac23f0605 | | GPT-5.2 | 71.3 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-397B | 70.8 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-235B | 68.9 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-235B | 66.8 | Not recorded | 6a2106eac10484bac23f0605 | | Kimi-K2 | 66.8 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-397B | 64.3 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-35B | 60.7 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-122B | 60.3 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-35B | 60.3 | Not recorded | 6a2106eac10484bac23f0605 | | Kimi-K2 | 55.3 | Not recorded | 6a2106eac10484bac23f0605 | | Kimi-K2 | 53.3 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-235B | 53.1 | Not recorded | 6a2106eac10484bac23f0605 | | Kimi-K2 | 51.8 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-235B | 50.9 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-Coder-480B | 50.5 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3.5-35B | 47.7 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-Coder-480B | 42.7 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-Coder-480B | 40.8 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-Coder-480B | 37.8 | Not recorded | 6a2106eac10484bac23f0605 | | Qwen3-Coder-480B | 33.8 | Not recorded | 6a2106eac10484bac23f0605 | ### GPT-5.2 Macro F1: 81.9. Result ID: 6a21799efbc373592c1ee49f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"codex"}. This record does not assert independent reproduction. ### GPT-5.2 Macro F1: 78.1. Result ID: 6a21799efbc373592c1ee4a0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3.5-122B Macro F1: 78. Result ID: 6a21799efbc373592c1ee489. [Reporting paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. ### GPT-5.2 Macro F1: 77.9. Result ID: 6a21799efbc373592c1ee49e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. ### Qwen3.5-397B Macro F1: 77.8. Result ID: 6a21799efbc373592c1ee494. [Reporting paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### Qwen3.5-122B Macro F1: 77.5. Result ID: 6a21799efbc373592c1ee48a. [Reporting paper]() Recorded evaluation configuration: {"Harness":"codex"}. This record does not assert independent reproduction. ### Qwen3.5-122B Macro F1: 76.4. Result ID: 6a21799efbc373592c1ee48c. [Reporting paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### Qwen3.5-122B Macro F1: 73. Result ID: 6a21799efbc373592c1ee48d. [Reporting paper]() Recorded evaluation configuration: {"Harness":"claude-code"}. This record does not assert independent reproduction. ### Qwen3.5-397B Macro F1: 72.7. Result ID: 6a21799efbc373592c1ee492. [Reporting paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. ### Qwen3.5-35B Macro F1: 71.8. Result ID: 6a21799efbc373592c1ee490. [Reporting paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### GPT-5.2 Macro F1: 71.3. Result ID: 6a21799efbc373592c1ee4a1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### Qwen3.5-397B Macro F1: 70.8. Result ID: 6a21799efbc373592c1ee493. [Reporting paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3-235B Macro F1: 68.9. Result ID: 6a21799efbc373592c1ee498. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3-235B Macro F1: 66.8. Result ID: 6a21799efbc373592c1ee499. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### Kimi-K2 Macro F1: 66.8. Result ID: 6a21799efbc373592c1ee49d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### Qwen3.5-397B Macro F1: 64.3. Result ID: 6a21799efbc373592c1ee495. [Reporting paper]() Recorded evaluation configuration: {"Harness":"claude-code"}. This record does not assert independent reproduction. ### Qwen3.5-35B Macro F1: 60.7. Result ID: 6a21799efbc373592c1ee48f. [Reporting paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3.5-122B Macro F1: 60.3. Result ID: 6a21799efbc373592c1ee48b. [Reporting paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3.5-35B Macro F1: 60.3. Result ID: 6a21799efbc373592c1ee491. [Reporting paper]() Recorded evaluation configuration: {"Harness":"claude-code"}. This record does not assert independent reproduction. ### Kimi-K2 Macro F1: 55.3. Result ID: 6a21799efbc373592c1ee49b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"codex"}. This record does not assert independent reproduction. ### Kimi-K2 Macro F1: 53.3. Result ID: 6a21799efbc373592c1ee49c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3-235B Macro F1: 53.1. Result ID: 6a21799efbc373592c1ee496. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. ### Kimi-K2 Macro F1: 51.8. Result ID: 6a21799efbc373592c1ee49a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. ### Qwen3-235B Macro F1: 50.9. Result ID: 6a21799efbc373592c1ee497. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"codex"}. This record does not assert independent reproduction. ### Qwen3-Coder-480B Macro F1: 50.5. Result ID: 6a21799efbc373592c1ee488. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"claude-code"}. This record does not assert independent reproduction. ### Qwen3.5-35B Macro F1: 47.7. Result ID: 6a21799efbc373592c1ee48e. [Reporting paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. ### Qwen3-Coder-480B Macro F1: 42.7. Result ID: 6a21799efbc373592c1ee485. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"codex"}. This record does not assert independent reproduction. ### Qwen3-Coder-480B Macro F1: 40.8. Result ID: 6a21799efbc373592c1ee486. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-2"}. This record does not assert independent reproduction. ### Qwen3-Coder-480B Macro F1: 37.8. Result ID: 6a21799efbc373592c1ee487. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"terminus-kira"}. This record does not assert independent reproduction. ### Qwen3-Coder-480B Macro F1: 33.8. Result ID: 6a21799efbc373592c1ee484. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Harness":"direct"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.