# Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering

Source: [Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering](<https://www.sota2.com/research/paper/68fa7c6c1e209131cea8dffa>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-17T05:00:05.253Z

Authors: Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata

Published: 2025-10-21T14:50:24.000Z

## Abstract

This study is the first to investigate LLM comprehension capabilities over long-context (LC), clinically relevant medical Question Answering (QA) beyond MCQA. Our comprehensive approach considers a range of settings based on content inclusion of varying size and relevance, LLM models of different capabilities and a variety of datasets across task formulations. We reveal insights on model size effects and their limitations, underlying memorization issues and the benefits of reasoning models, while demonstrating the value and challenges of leveraging the full long patient's context. Importantly, we examine the effect of Retrieval Augmented Generation (RAG) on medical LC comprehension, showcasing best settings in single versus multi-document QA datasets. We shed light into some of the evaluation aspects using a multi-faceted approach uncovering common metric challenges. Our quantitative analysis reveals challenging cases where RAG excels while still showing limitations in cases requiring temporal reasoning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.18691>)

[PDF](<https://arxiv.org/pdf/2510.18691.pdf>)