# Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

Source: [Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?](<https://www.sota2.com/research/paper/6a2a414fd33dfb23e7839549>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-12T05:00:14.585Z

Authors: Antoni Lasik, Jakub Pokrywka, {\\L}ukasz Grzybowski, Jeremi Ignacy Kaczmarek, Gabriela Korza\\'nska, Janusz \\'Swieczkowski-Feiz, Oskar Pastuszek, Paulina Hoffman, Jakub Tomasz D\\k{a}browski, Wojciech Kusa

Published: 2026-06-10T15:52:24.000Z

## Abstract

Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical ability due to guessing strategies and answer biases. To address these limitations, we introduce an expanded and more challenging benchmark based on Polish medical exams, adding over 15,000 questions, two new domains, and four structural modifications that reduce MCQA-specific artifacts and better test reasoning. We evaluate 21 LLMs and show that evaluation design strongly affects results. Under our harder setup, the best model (Qwen3.5-122B) drops by 28.4 and 31 pp on English and Polish exams, respectively. Despite low evidence of data contamination, standard MCQA scores do not reliably reflect true medical competence. To facilitate further research, we make our benchmark publicly available.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.12250>)

[PDF](<https://arxiv.org/pdf/2606.12250.pdf>)
