# Privacy Extraction on MemPrivacy-Bench v1 (test) Source: [Privacy Extraction on MemPrivacy-Bench v1 (test)]() Collection index: [llms.txt]() Data updated: 2026-05-12T12:24:57.849Z Evaluation of privacy extraction performance on the test split of the MemPrivacy-Bench v1 dataset, measured by F1 score, Precision, Recall, and inference time. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Privacy Extraction | | Dataset (including any stated split) | MemPrivacy-Bench v1 (test) | | Metric | F1 Score | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [F1 Score](): Higher is better; unit: PERCENT. - [Precision](): Higher is better; unit: PERCENT. - [Recall](): Higher is better; unit: PERCENT. - [Inference Time (s)](): Lower is better; unit: TIME. ## Recorded results | Method | F1 Score | Result date | Report paper | | --- | --- | --- | --- | | MemPrivacy-4B-RL | 85.97 | Not recorded | 6a02b46cdbc836b87d824b78 | | MemPrivacy-4B-SFT | 85.64 | Not recorded | 6a02b46cdbc836b87d824b78 | | MemPrivacy-0.6B-RL | 84.66 | Not recorded | 6a02b46cdbc836b87d824b78 | | MemPrivacy-1.7B-RL | 84.54 | Not recorded | 6a02b46cdbc836b87d824b78 | | MemPrivacy-1.7B-SFT | 84.22 | Not recorded | 6a02b46cdbc836b87d824b78 | | MemPrivacy-0.6B-SFT | 83.09 | Not recorded | 6a02b46cdbc836b87d824b78 | | Gemini-3.1-Pro | 78.41 | Not recorded | 6a02b46cdbc836b87d824b78 | | DeepSeek-V3.2-Think | 75.04 | Not recorded | 6a02b46cdbc836b87d824b78 | | GLM-5 | 70.1 | Not recorded | 6a02b46cdbc836b87d824b78 | | Kimi-K2 | 69.35 | Not recorded | 6a02b46cdbc836b87d824b78 | | GPT-5.2 | 68.99 | Not recorded | 6a02b46cdbc836b87d824b78 | | DeepSeek-V3.1-Terminus | 66.01 | Not recorded | 6a02b46cdbc836b87d824b78 | | DeepSeek-V3.2 | 65.91 | Not recorded | 6a02b46cdbc836b87d824b78 | | LongCat-Flash-Chat | 64.82 | Not recorded | 6a02b46cdbc836b87d824b78 | | Qwen3-4B | 59.34 | Not recorded | 6a02b46cdbc836b87d824b78 | | OpenAI-Privacy-Filter | 35.5 | Not recorded | 6a02b46cdbc836b87d824b78 | | Qwen3-1.7B | 30.83 | Not recorded | 6a02b46cdbc836b87d824b78 | | Qwen3-0.6B | 26.43 | Not recorded | 6a02b46cdbc836b87d824b78 | ### MemPrivacy-4B-RL F1 Score: 85.97. Result ID: 6a031bfc47f34ce924e74165. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Model Scale":"4B","Training Stage":"RL"}. This record does not assert independent reproduction. ### MemPrivacy-4B-SFT F1 Score: 85.64. Result ID: 6a031bfc47f34ce924e74162. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Model Scale":"4B","Training Stage":"SFT"}. This record does not assert independent reproduction. ### MemPrivacy-0.6B-RL F1 Score: 84.66. Result ID: 6a031bfc47f34ce924e74163. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Model Scale":"0.6B","Training Stage":"RL"}. This record does not assert independent reproduction. ### MemPrivacy-1.7B-RL F1 Score: 84.54. Result ID: 6a031bfc47f34ce924e74164. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Model Scale":"1.7B","Training Stage":"RL"}. This record does not assert independent reproduction. ### MemPrivacy-1.7B-SFT F1 Score: 84.22. Result ID: 6a031bfc47f34ce924e74161. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Model Scale":"1.7B","Training Stage":"SFT"}. This record does not assert independent reproduction. ### MemPrivacy-0.6B-SFT F1 Score: 83.09. Result ID: 6a031bfc47f34ce924e74160. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Model Scale":"0.6B","Training Stage":"SFT"}. This record does not assert independent reproduction. ### Gemini-3.1-Pro F1 Score: 78.41. Result ID: 6a031bfc47f34ce924e7415e. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### DeepSeek-V3.2-Think F1 Score: 75.04. Result ID: 6a031bfc47f34ce924e7415d. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### GLM-5 F1 Score: 70.1. Result ID: 6a031bfc47f34ce924e7415c. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### Kimi-K2 F1 Score: 69.35. Result ID: 6a031bfc47f34ce924e7415b. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### GPT-5.2 F1 Score: 68.99. Result ID: 6a031bfc47f34ce924e7415a. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### DeepSeek-V3.1-Terminus F1 Score: 66.01. Result ID: 6a031bfc47f34ce924e74158. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### DeepSeek-V3.2 F1 Score: 65.91. Result ID: 6a031bfc47f34ce924e74159. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### LongCat-Flash-Chat F1 Score: 64.82. Result ID: 6a031bfc47f34ce924e74157. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models"}. This record does not assert independent reproduction. ### Qwen3-4B F1 Score: 59.34. Result ID: 6a031bfc47f34ce924e74156. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models","Model Scale":"4B"}. This record does not assert independent reproduction. ### OpenAI-Privacy-Filter F1 Score: 35.5. Result ID: 6a031bfc47f34ce924e7415f. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Privacy Models","Type":"Token-classification"}. This record does not assert independent reproduction. ### Qwen3-1.7B F1 Score: 30.83. Result ID: 6a031bfc47f34ce924e74155. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models","Model Scale":"1.7B"}. This record does not assert independent reproduction. ### Qwen3-0.6B F1 Score: 26.43. Result ID: 6a031bfc47f34ce924e74154. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"General Models","Model Scale":"0.6B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.