Legal Question Answering on Japanese Bar Examination 2024 (Reiwa 6)
49.35Overall AccuracyOurs
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| OursFine-tuning Dataset=Dataset introduced in this paper, Self-Verification=true2026.01 | 49.35 | 94.7 | 94 | 96 | 22.3 | 42.3 | 30 | |
| OursFine-tuning Dataset=Dataset introduced in this paper, Self-Verification=false2026.01 | 46.75 | 92.3 | 91 | 93 | 20.3 | 42 | 30 | |
| Fine-Tuned w/ JBE-QAFine-tuning Dataset=JBE-QA, Self-Verification=true2026.01 | 42.26 | 80.7 | 78 | 82 | 21 | 32.7 | 27 | |
| BasePrompting Strategy=Few-Shot, Self-Verification=true2026.01 | 41.56 | 76.3 | 76 | 77 | 9.7 | 36.7 | 30 | |
| BasePrompting Strategy=Zero-Shot, Self-Verification=false2026.01 | 40.36 | 67 | 65 | 68 | 8 | 32 | 27 | |
| Multi-AgentMulti-Agent Configuration=same model for all agents2026.01 | 40.26 | 75.7 | 74 | 79 | 19.3 | 30.7 | 25.7 | |
| Multi-AgentMulti-Agent Configuration=separately fine-tuned models2026.01 | 39.69 | 71 | 66 | 77 | 12.7 | 34.7 | 25.6 | |
| BasePrompting Strategy=Few-Shot, Self-Verification=false2026.01 | 38.96 | 68.3 | 63 | 71 | 8 | 33.3 | 27 | |
| Fine-Tuned w/ JBE-QAFine-tuning Dataset=JBE-QA, Self-Verification=false2026.01 | 37.66 | 64 | 62 | 66 | 8 | 30 | 26 |