# Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity Source: [Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity]() Collection index: [llms.txt]() Data updated: 2026-05-09T05:00:12.752Z Authors: Florian A. D. Burnat, Brittany I. Davidson Published: 2026-05-07T14:23:31.000Z ## Abstract Safety benchmarks are routinely treated as evidence about how a language model will behave once deployed, but this inference is fragile if behavior depends on whether a prompt looks like an evaluation. We define evaluation-context divergence as an observable within-item change in behavior induced by framing a fixed task as an evaluation, a live deployment interaction, or a neutral request, and present a paired-prompt protocol that measures it in open-weight LLMs while controlling for paraphrase variation, benchmark familiarity, and judge framing-sensitivity. Across five instruction-tuned checkpoints from four open-weight families plus a matched OLMo-3 base/instruct ablation ($20$ paired items, $840$ generations per checkpoint), we find striking heterogeneity. OLMo-3-Instruct alone is eval-cautious -- evaluation framing raises refusal vs. neutral by $11.8$pp ($p=0.007$) and reduces harmful compliance vs. deployment by $3.6$pp ($p=0.024$, $0/20$ items inverted) -- while Mistral-Small-3.2, Phi-3.5-mini, and Llama-3.1-8B are deployment-cautious}, with marginal eval-vs-deployment refusal effects of $-9$ to $-20$pp. The matched OLMo-3 base also exhibits the deployment-cautious pattern, identifying alignment as the inversion stage; within Llama-3.1, the $70$B model preserves direction with attenuated magnitude, ruling out a simple \`\`small-model effect that reverses at scale.'' One caveat: the cross-family heterogeneity is judge-dependent. Re-judging with a different-family safety classifier (Llama-Guard-3-8B) preserves the within-OLMo eval-cautious direction but flattens the cross-family contrast, indicating that the two judges operationalize distinct constructs. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Harmful Compliance Analysis on Paired safety frame contrast items n=20 paired items per model (test)](): Harmful compliance analysis; Paired safety frame contrast items n=20 paired items per model (test); Harm Delta (eval-dep): -0.036. - [Refusal behavior analysis on Paired safety frame contrast items n=20 per model (test)](): Refusal behavior analysis; Paired safety frame contrast items n=20 per model (test); Refusal Delta (eval vs dep): 0.043. - [Harmful Compliance Prediction on Locked GLM (test)](): Harmful Compliance Prediction; Locked GLM (test); Beta: not recorded. - [Refusal Prediction on Locked GLM n=4,200, 20 item clusters (test)](): Refusal Prediction; Locked GLM n=4,200, 20 item clusters (test); Beta Coefficient: not recorded. ## Original sources and implementation [Original paper]() [PDF]()