Legal Domain Fabrication and Factuality Analysis on JURIDICO-FCR 75 tasks
0.11Average FCRGPT-5.2 (gpt-5.2-2025-12-11)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-5.2 (gpt-5.2-2025-12-11)Paradigm=Advanced RAG2026.01 | 0.11 | 0.1 | 1.1 | 0.5 | |
| Gemini 3.0 Pro (gemini-3.0-pro)Paradigm=Advanced RAG2026.01 | 0.12 | 0.11 | 1.4 | 0.6 | |
| GPT-4.1 (gpt-4.1-2025-04-14)Paradigm=Advanced RAG2026.01 | 0.13 | 0.11 | 1.6 | 0.8 | |
| DeepSeek V3.1 Terminus (deepseek-v3.1-terminus)Paradigm=Advanced RAG2026.01 | 0.13 | 0.12 | 1.7 | 1 | |
| Kimi K2 Thinking (Kimi-K2-Thinking)Paradigm=Advanced RAG2026.01 | 0.14 | 0.12 | 2 | 1.1 | |
| Claude 4.5 Sonnet (claude-sonnet-4-5...)Paradigm=Advanced RAG2026.01 | 0.14 | 0.13 | 2.4 | 1.2 | |
| Qwen3 235B A22B Instruct (Qwen3-235B-A22B...)Paradigm=Advanced RAG2026.01 | 0.15 | 0.13 | 2.2 | 1.3 | |
| Gemma 3 27B (gemma-3-27b-it)Paradigm=Advanced RAG2026.01 | 0.15 | 0.14 | 2.6 | 1.4 | |
| Llama 4 Maverick (Llama-4-Maverick...)Paradigm=Advanced RAG2026.01 | 0.16 | 0.14 | 2.7 | 1.5 | |
| Mistral Small 24B (Mistral-Small-24B...)Paradigm=Advanced RAG2026.01 | 0.17 | 0.15 | 3.1 | 1.7 | |
| Gemini Flash Latest (gemini-flash-preview...)Paradigm=Advanced RAG2026.01 | 0.18 | 0.15 | 3.3 | 1.8 | |
| Claude 4.1 Opus (claude-opus-4-1...)Paradigm=Advanced RAG2026.01 | 0.19 | 0.16 | 3.5 | 1.9 | |
| GPT-5.2 (gpt-5.2-2025-12-11)Paradigm=Canonical RAG2026.01 | 4.2 | 3.1 | 10 | 6.2 | |
| Gemini 3.0 Pro (gemini-3.0-pro)Paradigm=Canonical RAG2026.01 | 4.9 | 3.3 | 11.5 | 7 | |
| GPT-4.1 (gpt-4.1-2025-04-14)Paradigm=Canonical RAG2026.01 | 5.3 | 4.8 | 13 | 8.5 | |
| Kimi K2 Thinking (Kimi-K2-Thinking)Paradigm=Canonical RAG2026.01 | 6.8 | 5.9 | 15.5 | 10.5 | |
| DeepSeek V3.1 Terminus (deepseek-v3.1-terminus)Paradigm=Canonical RAG2026.01 | 7.1 | 5.2 | 14 | 9 | |
| Claude 4.5 Sonnet (claude-sonnet-4-5...)Paradigm=Canonical RAG2026.01 | 8.1 | 6.2 | 15 | 11.5 | |
| Qwen3 235B A22B Instruct (Qwen3-235B-A22B...)Paradigm=Canonical RAG2026.01 | 8.9 | 6.5 | 16 | 11 | |
| Llama 4 Maverick (Llama-4-Maverick...)Paradigm=Canonical RAG2026.01 | 9.5 | 7.8 | 18 | 13.5 | |
| Gemma 3 27B (gemma-3-27b-it)Paradigm=Canonical RAG2026.01 | 10.8 | 7.2 | 17 | 12.5 | |
| Mistral Small 24B (Mistral-Small-24B...)Paradigm=Canonical RAG2026.01 | 12 | 8.5 | 19 | 14 | |
| Gemini Flash Latest (gemini-flash-preview...)Paradigm=Canonical RAG2026.01 | 13.2 | 9.1 | 20.5 | 15.5 | |
| Claude 4.1 Opus (claude-opus-4-1...)Paradigm=Canonical RAG2026.01 | 14.8 | 9.8 | 21 | 16.5 | |
| GPT-5.2 (gpt-5.2-2025-12-11)Paradigm=Direct2026.01 | 15.1 | 8.2 | 35.5 | 23 | |
| Gemini 3.0 Pro (gemini-3.0-pro)Paradigm=Direct2026.01 | 16.8 | 8.5 | 38 | 25.5 | |
| GPT-4.1 (gpt-4.1-2025-04-14)Paradigm=Direct2026.01 | 18.5 | 10.9 | 43 | 27 | |
| Kimi K2 Thinking (Kimi-K2-Thinking)Paradigm=Direct2026.01 | 21.9 | 13.2 | 50 | 36 | |
| DeepSeek V3.1 Terminus (deepseek-v3.1-terminus)Paradigm=Direct2026.01 | 22.4 | 11.5 | 48 | 32 | |
| Qwen3 235B A22B Instruct (Qwen3-235B-A22B...)Paradigm=Direct2026.01 | 26.5 | 14.8 | 53 | 37 | |
| Claude 4.5 Sonnet (claude-sonnet-4-5...)Paradigm=Direct2026.01 | 28.2 | 15.5 | 59 | 41 | |
| Gemma 3 27B (gemma-3-27b-it)Paradigm=Direct2026.01 | 30.2 | 18.1 | 59 | 42 | |
| Llama 4 Maverick (Llama-4-Maverick...)Paradigm=Direct2026.01 | 31 | 16.9 | 62 | 44 | |
| Mistral Small 24B (Mistral-Small-24B...)Paradigm=Direct2026.01 | 34.5 | 19.2 | 64 | 47 | |
| Gemini Flash Latest (gemini-flash-preview...)Paradigm=Direct2026.01 | 36.8 | 20.9 | 67 | 50 | |
| Claude 4.1 Opus (claude-opus-4-1...)Paradigm=Direct2026.01 | 40 | 21.8 | 69 | 52.5 |