# LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications

Source: [LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications](<https://www.sota2.com/research/paper/68fa72f91e209131cea59aca>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:58:23.455Z

Authors: Saranya Krishnamoorthy, Ayush Singh, Shabnam Tafreshi

Published: 2024-04-25T02:25:35.000Z

## Abstract

Electronic health records (EHR) even though a boon for healthcare practitioners, are growing convoluted and longer every day. Sifting around these lengthy EHRs is taxing and becomes a cumbersome part of physician-patient interaction. Several approaches have been proposed to help alleviate this prevalent issue either via summarization or sectioning, however, only a few approaches have truly been helpful in the past. With the rise of automated methods, machine learning (ML) has shown promise in solving the task of identifying relevant sections in EHR. However, most ML methods rely on labeled data which is difficult to get in healthcare. Large language models (LLMs) on the other hand, have performed impressive feats in natural language processing (NLP), that too in a zero-shot manner, i.e. without any labeled data. To that end, we propose using LLMs to identify relevant section headers. We find that GPT-4 can effectively solve the task on both zero and few-shot settings as well as segment dramatically better than state-of-the-art methods. Additionally, we also annotate a much harder real world dataset and find that GPT-4 struggles to perform well, alluding to further research and harder benchmarks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Section Identification on i2b2 Corpus 2010](<https://www.sota2.com/research/sota/section-identification-on-i2b2-corpus-2010.md>): Section Identification; i2b2 Corpus 2010; Accuracy: 93.03.

- [Section Identification on MedSecId Corpus](<https://www.sota2.com/research/sota/section-identification-on-medsecid-corpus.md>): Section Identification; MedSecId Corpus; Accuracy: 96.86.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.16294>)

[PDF](<https://arxiv.org/pdf/2404.16294.pdf>)

[Implementation](<https://github.com/inQbator-eviCore/LLM_section_identifiers>)
