# Rule-Inconsistency Task Classification on RIT (Rule-Inconsistency Tasks) Source: [Rule-Inconsistency Task Classification on RIT (Rule-Inconsistency Tasks)]() Collection index: [llms.txt]() Data updated: 2026-02-26T06:08:22.143Z Evaluation of model performance on the Rule-Inconsistency Tasks (RIT) dataset, focusing on Rule-Inconsistency Task Classification. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Rule-Inconsistency Task Classification | | Dataset (including any stated split) | RIT (Rule-Inconsistency Tasks) | | Metric | WAC | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 30 of 30 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [WAC](): Higher is better; unit: PERCENT. - [SAC](): Higher is better; unit: PERCENT. - [WTC](): Higher is better; unit: PERCENT. - [STC](): Higher is better; unit: PERCENT. - [WCC](): Higher is better; unit: PERCENT. - [SCC](): Higher is better; unit: PERCENT. - [Total Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | WAC | Result date | Report paper | | --- | --- | --- | --- | | Llama-8b | 80.49 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-1.5-Pro | 75.61 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 75.61 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 67.07 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 64.63 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 56.1 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 54.88 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 53.66 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 52.44 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 51.22 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 51.22 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 51.22 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 50 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-1.5-Pro | 47.56 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 47.56 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 46.34 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-1.5-Pro | 43.9 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 32.93 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 29.27 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 29.27 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 28.05 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 25.61 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 25.61 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 18.29 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 13.41 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 10.98 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 4.88 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 4.88 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 4.88 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 3.66 | Not recorded | 695b456353fdd5c361742f15 | ### Llama-8b WAC: 80.49. Result ID: 6996f7605698665ec4441b25. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Zero-Shot"}. This record does not assert independent reproduction. ### Gemini-1.5-Pro WAC: 75.61. Result ID: 6996f7605698665ec4441b2e. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Zero-Shot"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 75.61. Result ID: 6996f7605698665ec4441b32. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"One-Shot"}. This record does not assert independent reproduction. ### Llama-70b WAC: 67.07. Result ID: 6996f7605698665ec4441b28. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Zero-Shot"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 64.63. Result ID: 6996f7605698665ec4441b33. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Two-Shot"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 56.1. Result ID: 6996f7605698665ec4441b31. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Zero-Shot"}. This record does not assert independent reproduction. ### Llama-8b WAC: 54.88. Result ID: 6996f7605698665ec4441b27. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Two-Shot"}. This record does not assert independent reproduction. ### Llama-8b WAC: 53.66. Result ID: 6996f7605698665ec4441b26. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"One-Shot"}. This record does not assert independent reproduction. ### Llama-70b WAC: 52.44. Result ID: 6996f7605698665ec4441b29. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"One-Shot"}. This record does not assert independent reproduction. ### Llama-70b WAC: 51.22. Result ID: 6996f7605698665ec4441b2a. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Two-Shot"}. This record does not assert independent reproduction. ### Llama-70b WAC: 51.22. Result ID: 6996f7605698665ec4441b52. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Zero-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro WAC: 51.22. Result ID: 6996f7605698665ec4441b58. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Zero-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Llama-70b WAC: 50. Result ID: 6996f7605698665ec4441b53. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"One-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Gemini-1.5-Pro WAC: 47.56. Result ID: 6996f7605698665ec4441b30. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Two-Shot"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro WAC: 47.56. Result ID: 6996f7605698665ec4441b59. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"One-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro WAC: 46.34. Result ID: 6996f7605698665ec4441b5a. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Two-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Gemini-1.5-Pro WAC: 43.9. Result ID: 6996f7605698665ec4441b2f. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"One-Shot"}. This record does not assert independent reproduction. ### GPT-4o WAC: 32.93. Result ID: 6996f7605698665ec4441b2b. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Zero-Shot"}. This record does not assert independent reproduction. ### GPT-4o WAC: 29.27. Result ID: 6996f7605698665ec4441b2d. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"Two-Shot"}. This record does not assert independent reproduction. ### GPT-4o WAC: 29.27. Result ID: 6996f7605698665ec4441b55. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Zero-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Llama-70b WAC: 28.05. Result ID: 6996f7605698665ec4441b54. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Two-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### GPT-4o WAC: 25.61. Result ID: 6996f7605698665ec4441b2c. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"One-Shot"}. This record does not assert independent reproduction. ### GPT-4o WAC: 25.61. Result ID: 6996f7605698665ec4441b56. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"One-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Llama-8b WAC: 18.29. Result ID: 6996f7605698665ec4441b4f. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Zero-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### GPT-4o WAC: 13.41. Result ID: 6996f7605698665ec4441b57. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Two-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Llama-8b WAC: 10.98. Result ID: 6996f7605698665ec4441b51. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Two-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Llama-8b WAC: 4.88. Result ID: 6996f7605698665ec4441b50. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"One-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 4.88. Result ID: 6996f7605698665ec4441b5c. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"One-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 4.88. Result ID: 6996f7605698665ec4441b5d. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Two-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. ### Deepseek-r1-7b WAC: 3.66. Result ID: 6996f7605698665ec4441b5b. [Reporting paper]() Recorded evaluation configuration: {"prompting\_strategy":"Zero-Shot","response\_constraint":"Single Response"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.