# Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA Source: [Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA]() Collection index: [llms.txt]() Data updated: 2026-03-06T05:00:05.534Z Authors: Ikram Belmadani, Oumaima El Khettari, Pac\\^ome Constant dit Beaufils, Richard Dufour, Benoit Favre Published: 2026-03-04T13:12:30.000Z ## Abstract Automatic evaluation of medical open-ended question answering (OEQA) remains challenging due to the need for expert annotations. We evaluate whether large language models (LLMs) can act as judges of semantic equivalence in French medical OEQA, comparing closed-access, general-purpose, and biomedical domain-adapted models. Our results show that LLM-based judgments are strongly influenced by the model that generated the answer, with agreement varying substantially across generators. Domain-adapted and large general-purpose models achieve the highest alignment with expert annotations. We further show that lightweight adaptation of a compact model using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) substantially improves performance and reduces generator sensitivity, even with limited data. Overall, our findings highlight the need for generator-aware evaluation and suggest that carefully adapted small models can support scalable evaluation in low-resource medical settings. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Binary equivalence judgment on French medical OEQA](): Binary equivalence judgment; French medical OEQA; Precision: 51.91. ## Original sources and implementation [Original paper]() [PDF]()