# Context-Aware Membership Inference Attacks against Pre-trained Large Language Models Source: [Context-Aware Membership Inference Attacks against Pre-trained Large Language Models]() Collection index: [llms.txt]() Data updated: 2025-10-26T05:39:30.468Z Authors: Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, Reza Shokri Published: 2024-09-11T01:56:35.000Z ## Abstract Membership Inference Attacks (MIAs) on pre-trained Large Language Models (LLMs) aim at determining if a data point was part of the model's training set. Prior MIAs that are built for classification models fail at LLMs, due to ignoring the generative nature of LLMs across token sequences. In this paper, we present a novel attack on pre-trained LLMs that adapts MIA statistical tests to the perplexity dynamics of subsequences within a data point. Our method significantly outperforms prior approaches, revealing context-dependent memorization patterns in pre-trained LLMs. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Membership Inference Attack on OPT-6.7B](): Membership Inference Attack; OPT-6.7B; TPR @ 1% FPR: 12.8. - [Membership Inference Attack on OLMo-7B](): Membership Inference Attack; Olmo-7B; TPR@1%FPR: 42.8. - [Membership Inference Attack on Pythia 6.9B](): Membership Inference Attack; Pythia 6.9B; TPR@1%FPR: 11.1. - [Membership Inference Attack on Llama-13B](): Membership Inference Attack; LLaMA-13B; TPR@1%FPR: 16.6. ## Original sources and implementation [Original paper]() [PDF]()