# Membership Inference on DM Math Pythia (train)

Source: [Membership Inference on DM Math Pythia (train)](<https://www.sota2.com/research/sota/membership-inference-on-dm-math-pythia-train>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:05:59.827Z

Evaluation of Membership Inference attack performance on the training split of the DM Math Pythia dataset, measured by TPR@1%FPR.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Membership Inference |
| Dataset (including any stated split) | DM Math Pythia (train) |
| Metric | TPR@1%FPR |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [TPR@1%FPR](<https://www.sota2.com/research/sota/membership-inference-on-dm-math-pythia-train.md?metric=TPR%401%25FPR>): Higher is better; unit: PERCENT.

## Recorded results

| Method | TPR@1%FPR | Result date | Report paper |
| --- | --- | --- | --- |
| AttenMIA (Perturbed) | 99.3 | Not recorded | 697846c1a2b28e761d87c68e |
| AttenMIA (Perturbed) | 98.9 | Not recorded | 697846c1a2b28e761d87c68e |
| AttenMIA (Perturbed) | 98.8 | Not recorded | 697846c1a2b28e761d87c68e |
| AttenMIA (Transitional) | 87.8 | Not recorded | 697846c1a2b28e761d87c68e |
| AttenMIA (Transitional) | 86.7 | Not recorded | 697846c1a2b28e761d87c68e |
| AttenMIA (Transitional) | 85.6 | Not recorded | 697846c1a2b28e761d87c68e |
| PPL attack | 51.7 | Not recorded | 697846c1a2b28e761d87c68e |
| reference attack | 50.6 | Not recorded | 697846c1a2b28e761d87c68e |
| reference attack | 49.4 | Not recorded | 697846c1a2b28e761d87c68e |
| reference attack | 47.2 | Not recorded | 697846c1a2b28e761d87c68e |
| RECALL | 44.2 | Not recorded | 697846c1a2b28e761d87c68e |
| PPL attack | 42.7 | Not recorded | 697846c1a2b28e761d87c68e |
| PPL attack | 34.8 | Not recorded | 697846c1a2b28e761d87c68e |
| RECALL | 31.2 | Not recorded | 697846c1a2b28e761d87c68e |
| RECALL | 22.1 | Not recorded | 697846c1a2b28e761d87c68e |

### AttenMIA (Perturbed)

TPR@1%FPR: 99.3. Result ID: 699918bb5698665ec47f157b.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

Recorded evaluation configuration: {"Model Size":"6.9b"}. This record does not assert independent reproduction.

### AttenMIA (Perturbed)

TPR@1%FPR: 98.9. Result ID: 699918bb5698665ec47f1579.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

Recorded evaluation configuration: {"Model Size":"1.4b"}. This record does not assert independent reproduction.

### AttenMIA (Perturbed)

TPR@1%FPR: 98.8. Result ID: 699918bb5698665ec47f157a.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

Recorded evaluation configuration: {"Model Size":"2.8b"}. This record does not assert independent reproduction.

### AttenMIA (Transitional)

TPR@1%FPR: 87.8. Result ID: 699918bb5698665ec47f1577.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

Recorded evaluation configuration: {"Model Size":"2.8b"}. This record does not assert independent reproduction.

### AttenMIA (Transitional)

TPR@1%FPR: 86.7. Result ID: 699918bb5698665ec47f1576.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

Recorded evaluation configuration: {"Model Size":"1.4b"}. This record does not assert independent reproduction.

### AttenMIA (Transitional)

TPR@1%FPR: 85.6. Result ID: 699918bb5698665ec47f1578.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

Recorded evaluation configuration: {"Model Size":"6.9b"}. This record does not assert independent reproduction.

### PPL attack

TPR@1%FPR: 51.7. Result ID: 699918bb5698665ec47f156d.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6a2b1e209131cea268e1.md>)

Recorded evaluation configuration: {"Model Size":"1.4b"}. This record does not assert independent reproduction.

### reference attack

TPR@1%FPR: 50.6. Result ID: 699918bb5698665ec47f1571.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6a2b1e209131cea268e1.md>)

Recorded evaluation configuration: {"Model Size":"2.8b"}. This record does not assert independent reproduction.

### reference attack

TPR@1%FPR: 49.4. Result ID: 699918bb5698665ec47f1570.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6a2b1e209131cea268e1.md>)

Recorded evaluation configuration: {"Model Size":"1.4b"}. This record does not assert independent reproduction.

### reference attack

TPR@1%FPR: 47.2. Result ID: 699918bb5698665ec47f1572.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6a2b1e209131cea268e1.md>)

Recorded evaluation configuration: {"Model Size":"6.9b"}. This record does not assert independent reproduction.

### RECALL

TPR@1%FPR: 44.2. Result ID: 699918bb5698665ec47f1573.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78b41e209131cea79dbb.md>)

Recorded evaluation configuration: {"Model Size":"1.4b"}. This record does not assert independent reproduction.

### PPL attack

TPR@1%FPR: 42.7. Result ID: 699918bb5698665ec47f156f.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6a2b1e209131cea268e1.md>)

Recorded evaluation configuration: {"Model Size":"6.9b"}. This record does not assert independent reproduction.

### PPL attack

TPR@1%FPR: 34.8. Result ID: 699918bb5698665ec47f156e.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6a2b1e209131cea268e1.md>)

Recorded evaluation configuration: {"Model Size":"2.8b"}. This record does not assert independent reproduction.

### RECALL

TPR@1%FPR: 31.2. Result ID: 699918bb5698665ec47f1574.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78b41e209131cea79dbb.md>)

Recorded evaluation configuration: {"Model Size":"2.8b"}. This record does not assert independent reproduction.

### RECALL

TPR@1%FPR: 22.1. Result ID: 699918bb5698665ec47f1575.

[Reporting paper](<https://www.sota2.com/research/paper/697846c1a2b28e761d87c68e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78b41e209131cea79dbb.md>)

Recorded evaluation configuration: {"Model Size":"6.9b"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
