Requirements Inconsistency Classification on RIT 3-category taxonomy v1.0 (test)
96.06ACDeepseek-r1-7b
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Deepseek-r1-7bPrompting Strategy=Zero-Shot, Response Constraint=Multiple responses2026.01 | 96.06 | 87.5 | 30 | 91.03 | |
| Deepseek-r1-7bPrompting Strategy=One-Shot, Response Constraint=Multiple responses2026.01 | 96.06 | 75 | 30 | 90.34 | |
| Llama-8bPrompting Strategy=Zero-Shot, Response Constraint=Multiple responses2026.01 | 92.91 | 100 | 20 | 88.28 | |
| Llama-8bPrompting Strategy=One-Shot, Response Constraint=Multiple responses2026.01 | 88.19 | 100 | 30 | 84.83 | |
| Llama-70bPrompting Strategy=Zero-Shot, Response Constraint=Multiple responses2026.01 | 73.23 | 50 | 30 | 68.97 | |
| Llama-70bPrompting Strategy=Two-Shot, Response Constraint=Multiple responses2026.01 | 66.14 | 62.5 | 50 | 64.83 | |
| Llama-70bPrompting Strategy=One-Shot, Response Constraint=Multiple responses2026.01 | 63.78 | 50 | 20 | 60 | |
| Gemini-2.5-ProPrompting Strategy=Zero-Shot, Response Constraint=Multiple responses2026.01 | 57.48 | 87.5 | 100 | 62.07 | |
| Llama-8bPrompting Strategy=Two-Shot, Response Constraint=Multiple responses2026.01 | 56.69 | 87.5 | 30 | 56.55 | |
| Gemini-2.5-ProPrompting Strategy=Two-Shot, Response Constraint=Multiple responses2026.01 | 55.91 | 100 | 100 | 61.38 | |
| Deepseek-r1-7bPrompting Strategy=Two-Shot, Response Constraint=Multiple responses2026.01 | 55.91 | 75 | 60 | 57.24 | |
| Gemini-2.5-ProPrompting Strategy=One-Shot, Response Constraint=Multiple responses2026.01 | 55.12 | 100 | 100 | 60.69 | |
| GPT-4oPrompting Strategy=Two-Shot, Response Constraint=Multiple responses2026.01 | 29.13 | 87.5 | 60 | 58.88 | |
| GPT-4oPrompting Strategy=Zero-Shot, Response Constraint=Multiple responses2026.01 | 25.2 | 75 | 90 | 63.4 | |
| GPT-4oPrompting Strategy=One-Shot, Response Constraint=Multiple responses2026.01 | 25.2 | 87.5 | 80 | 64.23 |