Business Rule Modeling on RAIR Longtail Hard (test)
71.5Accuracy@2LORE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| LOREModel Category=Proposed Method2025.12 | 71.5 | 58.2 | 46 | |
| GPT-5Model Category=prompting-based models2025.12 | 68.1 | 43.5 | 40.7 | |
| vanilla SFTModel Category=Supervised Fine-Tuning2025.12 | 67.1 | 54.2 | 41.3 | |
| Gemini 2.5 ProModel Category=prompting-based models2025.12 | 62.7 | 48.1 | 39.2 | |
| Qwen3-235B-InstructModel Category=prompting-based models2025.12 | 60.9 | 38.1 | 35.9 | |
| Qwen3-30B-InstructModel Category=prompting-based models2025.12 | 58.7 | 43.1 | 33.3 | |
| Qwen3-235B-ThinkingModel Category=prompting-based models2025.12 | 58.5 | 45.1 | 36.7 | |
| Qwen3-4B-ThinkingModel Category=prompting-based models2025.12 | 58.4 | 41.9 | 35.7 | |
| Qwen3-30B-ThinkingModel Category=prompting-based models2025.12 | 58.2 | 44.8 | 36.2 | |
| Llama3.1-70B-InstructModel Category=prompting-based models2025.12 | 54.7 | 37.8 | 29.5 | |
| Qwen2.5-7B(base)Model Category=prompting-based models2025.12 | 53.1 | 38.2 | 31.2 | |
| Llama3.1-8B-InstructModel Category=prompting-based models2025.12 | 52.5 | 24.8 | 20 |