# Membership Inference Attack on arXiv Pythia Source: [Membership Inference Attack on arXiv Pythia]() Collection index: [llms.txt]() Data updated: 2026-02-26T06:00:24.070Z Evaluation of a model's susceptibility to Membership Inference Attacks on the arXiv Pythia dataset, measured by metrics such as ROC AUC. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Membership Inference Attack | | Dataset (including any stated split) | arXiv Pythia | | Metric | ROC AUC | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 36 of 36 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [ROC AUC](): Higher is better; unit: PERCENT. ## Recorded results | Method | ROC AUC | Result date | Report paper | | --- | --- | --- | --- | | AttenMIA (Perturbed) | 94 | Not recorded | 697846c1a2b28e761d87c68e | | AttenMIA (Perturbed) | 94 | Not recorded | 697846c1a2b28e761d87c68e | | AttenMIA (Perturbed) | 93 | Not recorded | 697846c1a2b28e761d87c68e | | AttenMIA (Transitional) | 81 | Not recorded | 697846c1a2b28e761d87c68e | | AttenMIA (Transitional) | 79 | Not recorded | 697846c1a2b28e761d87c68e | | AttenMIA (Transitional) | 78 | Not recorded | 697846c1a2b28e761d87c68e | | neighborhood attack | 69 | Not recorded | 697846c1a2b28e761d87c68e | | neighborhood attack | 69 | Not recorded | 697846c1a2b28e761d87c68e | | neighborhood attack | 69 | Not recorded | 697846c1a2b28e761d87c68e | | reference attack | 67 | Not recorded | 697846c1a2b28e761d87c68e | | reference attack | 67 | Not recorded | 697846c1a2b28e761d87c68e | | PPL attack | 66 | Not recorded | 697846c1a2b28e761d87c68e | | PPL attack | 66 | Not recorded | 697846c1a2b28e761d87c68e | | zlib attack | 66 | Not recorded | 697846c1a2b28e761d87c68e | | zlib attack | 65 | Not recorded | 697846c1a2b28e761d87c68e | | PPL attack | 64 | Not recorded | 697846c1a2b28e761d87c68e | | reference attack | 64 | Not recorded | 697846c1a2b28e761d87c68e | | zlib attack | 64 | Not recorded | 697846c1a2b28e761d87c68e | | MIN-K% PROB | 62 | Not recorded | 697846c1a2b28e761d87c68e | | MIN-K% PROB | 62 | Not recorded | 697846c1a2b28e761d87c68e | | MIN-K% PROB | 61 | Not recorded | 697846c1a2b28e761d87c68e | | PETAL | 58 | Not recorded | 697846c1a2b28e761d87c68e | | PETAL | 58 | Not recorded | 697846c1a2b28e761d87c68e | | PETAL | 57 | Not recorded | 697846c1a2b28e761d87c68e | | BT attack | 55 | Not recorded | 697846c1a2b28e761d87c68e | | RECALL | 55 | Not recorded | 697846c1a2b28e761d87c68e | | RECALL | 53 | Not recorded | 697846c1a2b28e761d87c68e | | RECALL | 53 | Not recorded | 697846c1a2b28e761d87c68e | | RS attack | 51 | Not recorded | 697846c1a2b28e761d87c68e | | BT attack | 51 | Not recorded | 697846c1a2b28e761d87c68e | | BT attack | 51 | Not recorded | 697846c1a2b28e761d87c68e | | WS attack | 50 | Not recorded | 697846c1a2b28e761d87c68e | | WS attack | 50 | Not recorded | 697846c1a2b28e761d87c68e | | WS attack | 50 | Not recorded | 697846c1a2b28e761d87c68e | | RS attack | 50 | Not recorded | 697846c1a2b28e761d87c68e | | RS attack | 48 | Not recorded | 697846c1a2b28e761d87c68e | ### AttenMIA (Perturbed) ROC AUC: 94. Result ID: 699918bb5698665ec47f1648. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### AttenMIA (Perturbed) ROC AUC: 94. Result ID: 699918bb5698665ec47f1649. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### AttenMIA (Perturbed) ROC AUC: 93. Result ID: 699918bb5698665ec47f1647. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### AttenMIA (Transitional) ROC AUC: 81. Result ID: 699918bb5698665ec47f1644. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### AttenMIA (Transitional) ROC AUC: 79. Result ID: 699918bb5698665ec47f1645. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### AttenMIA (Transitional) ROC AUC: 78. Result ID: 699918bb5698665ec47f1646. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### neighborhood attack ROC AUC: 69. Result ID: 699918bb5698665ec47f162f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### neighborhood attack ROC AUC: 69. Result ID: 699918bb5698665ec47f1630. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### neighborhood attack ROC AUC: 69. Result ID: 699918bb5698665ec47f1631. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### reference attack ROC AUC: 67. Result ID: 699918bb5698665ec47f1629. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### reference attack ROC AUC: 67. Result ID: 699918bb5698665ec47f162b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### PPL attack ROC AUC: 66. Result ID: 699918bb5698665ec47f1626. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### PPL attack ROC AUC: 66. Result ID: 699918bb5698665ec47f1628. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### zlib attack ROC AUC: 66. Result ID: 699918bb5698665ec47f162e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### zlib attack ROC AUC: 65. Result ID: 699918bb5698665ec47f162c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### PPL attack ROC AUC: 64. Result ID: 699918bb5698665ec47f1627. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### reference attack ROC AUC: 64. Result ID: 699918bb5698665ec47f162a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### zlib attack ROC AUC: 64. Result ID: 699918bb5698665ec47f162d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### MIN-K% PROB ROC AUC: 62. Result ID: 699918bb5698665ec47f1632. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### MIN-K% PROB ROC AUC: 62. Result ID: 699918bb5698665ec47f1634. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### MIN-K% PROB ROC AUC: 61. Result ID: 699918bb5698665ec47f1633. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### PETAL ROC AUC: 58. Result ID: 699918bb5698665ec47f163e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### PETAL ROC AUC: 58. Result ID: 699918bb5698665ec47f1640. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### PETAL ROC AUC: 57. Result ID: 699918bb5698665ec47f163f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### BT attack ROC AUC: 55. Result ID: 699918bb5698665ec47f163d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### RECALL ROC AUC: 55. Result ID: 699918bb5698665ec47f1643. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### RECALL ROC AUC: 53. Result ID: 699918bb5698665ec47f1641. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### RECALL ROC AUC: 53. Result ID: 699918bb5698665ec47f1642. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### RS attack ROC AUC: 51. Result ID: 699918bb5698665ec47f163a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### BT attack ROC AUC: 51. Result ID: 699918bb5698665ec47f163b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### BT attack ROC AUC: 51. Result ID: 699918bb5698665ec47f163c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### WS attack ROC AUC: 50. Result ID: 699918bb5698665ec47f1635. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### WS attack ROC AUC: 50. Result ID: 699918bb5698665ec47f1636. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. ### WS attack ROC AUC: 50. Result ID: 699918bb5698665ec47f1637. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"6.9b"}. This record does not assert independent reproduction. ### RS attack ROC AUC: 50. Result ID: 699918bb5698665ec47f1638. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"1.4b"}. This record does not assert independent reproduction. ### RS attack ROC AUC: 48. Result ID: 699918bb5698665ec47f1639. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model size":"2.8b"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.