# Reliability by design: quantifying and eliminating fabrication risk in LLMs. From generative to consultative AI: a comparative analysis in the legal domain and lessons for high-stakes knowledge bases

Source: [Reliability by design: quantifying and eliminating fabrication risk in LLMs. From generative to consultative AI: a comparative analysis in the legal domain and lessons for high-stakes knowledge bases](<https://www.sota2.com/research/paper/69730060b2be0ed85a78c834>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-24T05:00:05.231Z

Authors: Alex Dantart

Published: 2026-01-21T21:26:42.000Z

## Abstract

This paper examines how to make large language models reliable for high-stakes legal work by reducing hallucinations. It distinguishes three AI paradigms: (1) standalone generative models ("creative oracle"), (2) basic retrieval-augmented systems ("expert archivist"), and (3) an advanced, end-to-end optimized RAG system ("rigorous archivist"). The authors introduce two reliability metrics -False Citation Rate (FCR) and Fabricated Fact Rate (FFR)- and evaluate 2,700 judicial-style answers from 12 LLMs across 75 legal tasks using expert, double-blind review. Results show that standalone models are unsuitable for professional use (FCR above 30%), while basic RAG greatly reduces errors but still leaves notable misgrounding. Advanced RAG, using techniques such as embedding fine-tuning, re-ranking, and self-correction, reduces fabrication to negligible levels (below 0.2%). The study concludes that trustworthy legal AI requires rigor-focused, retrieval-based architectures emphasizing verification and traceability, and provides an evaluation framework applicable to other high-risk domains.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Legal Domain Fabrication and Factuality Analysis on JURIDICO-FCR 75 tasks](<https://www.sota2.com/research/sota/legal-domain-fabrication-and-factuality-analysis-on-juridico-fcr-75-tasks.md>): Legal Domain Fabrication and Factuality Analysis; JURIDICO-FCR 75 tasks; Average FCR: 0.11.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.15476>)

[PDF](<https://arxiv.org/pdf/2601.15476.pdf>)