Citation Grounding Measures the Oracle: Graph Coverage Determines Reported LLM Hallucination Rates in Law
About
Verifying LLM-generated legal citations against a graph of citations extracted from real court decisions is an appealing way to measure hallucination at scale: no annotators, no reference answers. We show that what such a metric reports is governed by the coverage of the graph it queries rather than by the model it evaluates, and that at the coverage where its verdicts become trustworthy it stops distinguishing models at all. We score 400 responses (100 Ukrainian legal queries x four commercial LLMs) against two snapshots of the same national citation graph, holding responses, extractor and metric fixed. Against a sparse snapshot (4.7e5 records) citation grounding ranges 0.791-0.855, apparently showing 15-21% of citations hallucinated. Against a dense snapshot of the same registry re-derived ten weeks later (3.3e8 records, 5.8e7 decisions) the identical responses score 0.989-0.999. Subsampling shows the cause is coverage: harvesting modelled as uniform record sampling, calibrated on nothing but the record count, reproduces the sparse scores to within 0.018 while knowing nothing about the models. Bootstrapping over the queries shows the other half, which no version of this work reported: no pair of systems is separable at 95% at any oracle size tested. The metric is caught between two failures. Sparse oracles discriminate, but what they discriminate is harvesting coverage; dense oracles are trustworthy and separate nothing. Densifying the graph, the obvious remedy for the first, produces the second. An independent legislation registry adjudicates: all 54 citations flagged by the sparse oracle name real articles, a 100% false-positive rate; of the four flagged by the dense oracle, two are fabrications, one a coverage gap, and one we cannot classify - it looks like a repealed provision, which is the argument in miniature.