# Performance of Large Language Models in Technical MRI Question Answering: A Comparative Study

Source: [Performance of Large Language Models in Technical MRI Question Answering: A Comparative Study](<https://www.sota2.com/research/paper/68fa489a1e209131ce930c80>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T04:33:06.473Z

Authors: Alan B McMillan

Published: 2024-11-19T05:29:58.000Z

## Abstract

Background: Advances in artificial intelligence, particularly large language models (LLMs), have the potential to enhance technical expertise in magnetic resonance imaging (MRI), regardless of operator skill or geographic location.   Methods: We assessed the accuracy of several LLMs in answering 570 technical MRI questions derived from a standardized review book. The questions spanned nine MRI topics, including Basic Principles, Image Production, and Safety. Closed-source models (e.g., OpenAI's o1 Preview, GPT-4o, GPT-4 Turbo, and Claude 3.5 Haiku) and open-source models (e.g., Phi 3.5 Mini, Llama 3.1, smolLM2) were tested. Models were queried using standardized prompts via the LangChain framework, and responses were graded against correct answers using an automated scoring protocol. Accuracy, defined as the proportion of correct answers, was the primary outcome.   Results: The closed-source o1 Preview model achieved the highest accuracy (94%), exceeding the random-guess baseline (26.5%). GPT-4o and o1 Mini scored 88%, and GPT-4 Turbo and Claude 3.5 Haiku each scored 84%. Among open-source models, Phi 3.5 Mini performed well, achieving 78% accuracy, comparable to several closed-source models. Accuracy was highest in Basic Principles and Instrumentation categories but lower in Image Weighting and Contrast, History, and Artifacts and Corrections.   Conclusions: LLMs exhibit high accuracy in addressing technical MRI questions, suggesting their potential to standardize and enhance MRI practice. These models may improve image quality and consistency across varied clinical environments. Further studies are needed to refine LLMs for clinical use and integrate them into MRI workflows.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.12238>)

[PDF](<https://arxiv.org/pdf/2411.12238.pdf>)