Privacy Appropriateness Classification on SPA
87.4F1 (Appropriateness)ARIEL
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ARIELModel=GPT-52025.12 | 87.4 | 94.2 | 3,810 | |
| ARIELModel=Gemini 2.5 Pro2025.12 | 87.2 | 94.2 | 3,797 | |
| ARIELModel=Claude Sonnet 4.52025.12 | 86.6 | 93.8 | 3,824 | |
| ICL (w Undet)Model=Claude Sonnet 4.52025.12 | 83.3 | 91.7 | 4,900 | |
| ICL (w Undet)Model=Gemini 2.5 Pro2025.12 | 82.4 | 92.2 | 4,613 | |
| ICLModel=Gemini 2.5 Pro2025.12 | 81.8 | 91 | 5,000 | |
| ICLModel=Claude Sonnet 4.52025.12 | 81.4 | 91 | 5,000 | |
| ICL (w Undet)Model=GPT-52025.12 | 80 | 91.1 | 4,921 | |
| ICLModel=GPT-52025.12 | 79 | 90.6 | 5,000 | |
| ARIELModel=Gemma 3 4B2025.12 | 75.5 | 88.4 | 3,680 | |
| ICLModel=Gemma 3 4B2025.12 | 64.1 | 84.4 | 5,000 | |
| ICL (w Undet)Model=Gemma 3 4B2025.12 | 62.5 | 84.2 | 4,999 | |
| privacy normsModel=Gemini 2.5 Pro2025.12 | 59.1 | 71.5 | — | |
| privacy normsModel=GPT-52025.12 | 58.4 | 71.1 | — | |
| privacy normsModel=Claude Sonnet 4.52025.12 | 57.7 | 73.6 | — | |
| zero-shotModel=Gemini 2.5 Pro2025.12 | 49.4 | 75.8 | — | |
| privacy normsModel=Gemma 3 4B2025.12 | 46.4 | 73.1 | — | |
| zero-shotModel=Gemma 3 4B2025.12 | 45.1 | 72.8 | — | |
| zero-shotModel=Claude Sonnet 4.52025.12 | 40 | 75 | — | |
| zero-shotModel=GPT-52025.12 | 27.1 | 78.3 | — |