# Requirements Inconsistency Classification on RIT 3-category taxonomy v1.0 (test) Source: [Requirements Inconsistency Classification on RIT 3-category taxonomy v1.0 (test)]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:21:47.996Z Evaluation of Requirements Inconsistency Classification performance on the test split of the RIT 3-category taxonomy v1.0 dataset, using metrics like AC, TC, CC, and Total Correct. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Requirements Inconsistency Classification | | Dataset (including any stated split) | RIT 3-category taxonomy v1.0 (test) | | Metric | AC | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 15 of 15 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [AC](): Higher is better; unit: PERCENT. - [TC](): Higher is better; unit: PERCENT. - [CC](): Higher is better; unit: PERCENT. - [Total Correct](): Higher is better; unit: PERCENT. ## Recorded results | Method | AC | Result date | Report paper | | --- | --- | --- | --- | | Deepseek-r1-7b | 96.06 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 96.06 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 92.91 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 88.19 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 73.23 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 66.14 | Not recorded | 695b456353fdd5c361742f15 | | Llama-70b | 63.78 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 57.48 | Not recorded | 695b456353fdd5c361742f15 | | Llama-8b | 56.69 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 55.91 | Not recorded | 695b456353fdd5c361742f15 | | Deepseek-r1-7b | 55.91 | Not recorded | 695b456353fdd5c361742f15 | | Gemini-2.5-Pro | 55.12 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 29.13 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 25.2 | Not recorded | 695b456353fdd5c361742f15 | | GPT-4o | 25.2 | Not recorded | 695b456353fdd5c361742f15 | ### Deepseek-r1-7b AC: 96.06. Result ID: 6996f7605698665ec4441b4c. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Zero-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Deepseek-r1-7b AC: 96.06. Result ID: 6996f7605698665ec4441b4d. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"One-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Llama-8b AC: 92.91. Result ID: 6996f7605698665ec4441b40. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Zero-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Llama-8b AC: 88.19. Result ID: 6996f7605698665ec4441b41. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"One-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Llama-70b AC: 73.23. Result ID: 6996f7605698665ec4441b43. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Zero-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Llama-70b AC: 66.14. Result ID: 6996f7605698665ec4441b45. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Two-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Llama-70b AC: 63.78. Result ID: 6996f7605698665ec4441b44. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"One-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro AC: 57.48. Result ID: 6996f7605698665ec4441b49. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Zero-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Llama-8b AC: 56.69. Result ID: 6996f7605698665ec4441b42. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Two-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro AC: 55.91. Result ID: 6996f7605698665ec4441b4b. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Two-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Deepseek-r1-7b AC: 55.91. Result ID: 6996f7605698665ec4441b4e. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Two-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro AC: 55.12. Result ID: 6996f7605698665ec4441b4a. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"One-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### GPT-4o AC: 29.13. Result ID: 6996f7605698665ec4441b48. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Two-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### GPT-4o AC: 25.2. Result ID: 6996f7605698665ec4441b46. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"Zero-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. ### GPT-4o AC: 25.2. Result ID: 6996f7605698665ec4441b47. [Reporting paper]() Recorded evaluation configuration: {"Prompting Strategy":"One-Shot","Response Constraint":"Multiple responses"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.