# What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA

Source: [What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA](<https://www.sota2.com/research/paper/6a15297adbd7150506f7e475>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-27T05:00:21.000Z

Authors: Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wan

Published: 2026-05-25T16:06:06.000Z

## Abstract

Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive. \\textbf{We find that the checker's \\emph{output distribution} during training, not its held-out accuracy, decides whether it provides trainable gradient.} We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent (Qwen2.5-7B, replicated on Qwen3-4B and Llama-3.1-8B) across four held-out medical QA benchmarks. Three diagnostic findings emerge. \\textbf{(i)} Signal collapse is log-prob-specific: LLM log-probability scoring labels over 97\\% of claims neutral -- collapsing the RL gradient to zero -- while a calibrated MedNLI classifier scores the same pairs non-degenerately. \\textbf{(ii)} Moderate signal beats strong signal on answer quality: a strong proprietary checker triggers a three-step reward-hacking cascade -- ultra-short answers, search avoidance, language collapse -- so a moderate-signal local classifier trains a higher-quality model (\\textbf{+12\\% BERTScore over zero-shot, no GPT dependency}). \\textbf{(iii)} Signal strength is policy-dependent: the same checker registers as moderate on one policy but strong on another without triggering the cascade end-state. We frame these as boundary conditions for verifier-as-reward systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-choice on MIRAGE](<https://www.sota2.com/research/sota/multi-choice-on-mirage.md>): Multi-Choice; Mirage; Accuracy: 58.3.

- [Short-fact on MedBrowseComp](<https://www.sota2.com/research/sota/short-fact-on-medbrowsecomp.md>): Short-fact; MedBrowseComp; Accuracy: 8.4.

- [Medical Question Answering on MedicationQA full n=674 (test)](<https://www.sota2.com/research/sota/medical-question-answering-on-medicationqa-full-n-674-test.md>): Medical Question Answering; MedicationQA full n=674 (test); F1 Score: 19.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.25988>)

[PDF](<https://arxiv.org/pdf/2605.25988.pdf>)