Contextual Integrity Reasoning on PrivacyLens
52.9LR (%)Qwen-14B-IT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Qwen-14B-ITCategory=Open Weights, Reasoning Strategy=Baseline2026.05 | 52.9 | 51.2 | 2.37 | |
| Qwen-7B-ITCategory=Open Weights, Reasoning Strategy=Baseline2026.05 | 50.3 | 52.4 | 1.99 | |
| Mistral-7B-ITCategory=Open Weights, Reasoning Strategy=Baseline2026.05 | 47.9 | 52.1 | 1.78 | |
| Qwen-7B-ITCategory=Open Weights, Reasoning Strategy=CI-CoT2026.05 | 44.8 | 45.7 | 2.13 | |
| Qwen-14B-ITCategory=Open Weights, Reasoning Strategy=CI-CoT2026.05 | 42.8 | 44.4 | 2.27 | |
| Gemini 2.5 ProCategory=Baseline LRMs, Reasoning Strategy=Baseline2026.05 | 37.3 | 38.2 | 2.84 | |
| Qwen-14B-ITCategory=Open Weights, Reasoning Strategy=CI-RL2026.05 | 33.9 | 34.4 | 2.3 | |
| Qwen-7B-ITCategory=Open Weights, Reasoning Strategy=CI-RL2026.05 | 33.7 | 33.9 | 2.08 | |
| Claude 3.7 S-TCategory=Baseline LRMs, Reasoning Strategy=Baseline2026.05 | 32 | 34.6 | 2.75 | |
| Mistral-7B-ITCategory=Open Weights, Reasoning Strategy=CI-RL2026.05 | 31.2 | 29.6 | 1.84 | |
| Claude 3.7 SCategory=Baseline LLMs, Reasoning Strategy=Baseline2026.05 | 30.4 | 35.9 | 2.49 | |
| Gemini 2.5 FCategory=Baseline LLMs, Reasoning Strategy=Baseline2026.05 | 29 | 30.8 | 2.75 | |
| Mistral-7B-ITCategory=Open Weights, Reasoning Strategy=CI-CoT2026.05 | 28.8 | 46.6 | 1.17 | |
| Gemini 2.5 ProCategory=Baseline LRMs, Reasoning Strategy=CI-CoT2026.05 | 25.3 | 26.9 | 2.72 | |
| Claude 3.7 SCategory=Baseline LLMs, Reasoning Strategy=CI-CoT2026.05 | 23.1 | 25.4 | 2.69 | |
| Llama-8B-ITCategory=Open Weights, Reasoning Strategy=CI-CoT2026.05 | 21.3 | 31.5 | 1.29 | |
| Claude 3.7 S-TCategory=Baseline LRMs, Reasoning Strategy=CI-CoT2026.05 | 20.1 | 22.6 | 2.63 | |
| Gemini 2.5 FCategory=Baseline LLMs, Reasoning Strategy=CI-CoT2026.05 | 19.7 | 24 | 2.31 | |
| Llama-8B-ITCategory=Open Weights, Reasoning Strategy=CI-RL2026.05 | 18.5 | 29.4 | 1.18 | |
| Llama-8B-ITCategory=Open Weights, Reasoning Strategy=Baseline2026.05 | 18.2 | 38.9 | 1.05 |