Multi-hop Privacy Leakage Reasoning on MosaicLeaks (test)
63.6Strict Chain SuccessGemma 4 31B IT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gemma 4 31B ITprivacy prompt=false2026.05 | 63.6 | 82 | 9.7 | 25.8 | 6.2 | 2.3 | |
| Gemma 4 31B ITprivacy prompt=true2026.05 | 60.8 | 80.9 | 1.9 | 14.9 | 1.3 | 0.4 | |
| GPT-OSS 20Bprivacy prompt=true2026.05 | 58.3 | 78.3 | 34.8 | 35.9 | 18.3 | 10.2 | |
| GPT-OSS 20Bprivacy prompt=false2026.05 | 52.4 | 73.1 | 42.9 | 42.2 | 21.8 | 14.6 | |
| StepFun-3.5-Flashprivacy prompt=true2026.05 | 51.6 | 75.2 | 39.9 | 54.6 | 17.7 | 14 | |
| StepFun-3.5-Flashprivacy prompt=false2026.05 | 51.4 | 74.8 | 39.6 | 53.8 | 17.5 | 14.1 | |
| Qwen3 8Bprivacy prompt=true2026.05 | 50.1 | 70.8 | 27.2 | 36.6 | 12.4 | 7.8 | |
| Qwen3-4B-Instructprivacy prompt=false2026.05 | 48.7 | 73.1 | 34 | 51.1 | 18.7 | 11.7 | |
| Qwen3 8Bprivacy prompt=false2026.05 | 48.4 | 69.7 | 33.4 | 45.8 | 15.4 | 10.4 | |
| Qwen3-4B-Instructprivacy prompt=true2026.05 | 44.5 | 71.3 | 25.5 | 39.2 | 14.4 | 7.9 | |
| Chroma Context-1privacy prompt=true, configuration=medium OpenRouter2026.05 | 25.6 | 51.1 | 17.9 | 23.5 | 5.6 | 3.2 | |
| Chroma Context-1privacy prompt=false, configuration=medium OpenRouter2026.05 | 15.2 | 36.7 | 19.1 | 25.3 | 5.7 | 3.3 |