# LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation Source: [LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation]() Collection index: [llms.txt]() Data updated: 2026-06-09T05:00:15.114Z Authors: Luk\\'a\\v{s} Eigler, Jind\\v{r}ich Libovick\\'y, David Hurych Published: 2026-03-10T09:15:19.000Z ## Abstract Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Machine Translation Evaluation on WMT 21](): Machine Translation Evaluation; WMT 21; Score (en-ha): 54.3. - [Machine Translation Evaluation on WMT 24](): Machine Translation Evaluation; WMT 24; Quality (cs-uk): 0.945. - [NLG Meta-evaluation on CUS-QA en cs](): NLG Meta-evaluation; CUS-QA en cs; Kendall Correlation: 0.73. - [NLG Meta-evaluation on CUS-QA en (sk)](): NLG Meta-evaluation; CUS-QA en (sk); Kendall Correlation: 0.661. - [NLG Meta-evaluation on CUS-QA en uk](): NLG Meta-evaluation; CUS-QA en uk; Kendall Correlation: 0.577. - [NLG Meta-evaluation on CUS-QA orig. (cs)](): NLG Meta-evaluation; CUS-QA orig. (cs); Kendall Correlation: 0.804. - [NLG Meta-evaluation on CUS-QA orig. (sk)](): NLG Meta-evaluation; CUS-QA orig. (sk); Kendall Correlation: 0.788. - [NLG Meta-evaluation on CUS-QA orig. (uk)](): NLG Meta-evaluation; CUS-QA orig. (uk); Kendall Correlation: 0.681. - [NLG Meta-evaluation on MOCHA](): NLG Meta-evaluation; MOCHA; Kendall Correlation: 0.72. - [NLG Meta-evaluation on RoSE](): NLG Meta-evaluation; RoSE; Kendall Correlation: 0.635. - [NLG Meta-evaluation on WMT en-ha 21](): NLG Meta-evaluation; WMT en-ha 21; Kendall Correlation: 0.386. - [NLG Meta-evaluation on WMT xh-zu 21](): NLG Meta-evaluation; WMT xh-zu 21; Kendall Correlation: 0.323. - [NLG Meta-evaluation on WMT 24 cs-uk](): NLG Meta-evaluation; WMT 24 cs-uk; Kendall Correlation: 0.825. - [NLG Meta-evaluation on WMT en-cs 24](): NLG Meta-evaluation; WMT en-cs 24; Kendall Correlation: 0.318. - [NLG Meta-evaluation on WMT en-is 24](): NLG Meta-evaluation; WMT en-is 24; Kendall Correlation: 0.36. - [Question Answering Evaluation on CUS-QA en](): Question Answering Evaluation; CUS-QA en; CS Metric: 91.7. - [Question Answering on CUS-QA orig.](): Question Answering Evaluation; CUS-QA orig.; CS: 95.6. - [Question Answering on MOCHA](): Question Answering Evaluation; MOCHA; Spearman Correlation: 0.872. - [Summarization Evaluation on RoSE](): Summarization Evaluation; RoSE; Spearman Correlation: 0.833. ## Original sources and implementation [Original paper]() [PDF]()