Variation error identification and evaluation on VariErr (Expanded)
63.7AccuracyQwen3
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3Setting=SFT-ind.2026.05 | 63.7 | 28.3 | 89.2 | 59 | 0.084 | 80.5 | 84.5 | 0.859 | 30 | |
| Qwen3Setting=SFT-shared2026.05 | 63.2 | 24.4 | 88.7 | 56.2 | 0.28 | 29.6 | 31.2 | 0.308 | 26.3 | |
| Qwen3Setting=CAPO-strict2026.05 | 62.7 | 28.1 | 89.1 | 58.2 | 0.081 | 82.3 | 86.7 | 0.888 | 32.8 | |
| Llama3.2Setting=SFT-shared2026.05 | 57.5 | 22.9 | 88.3 | 55.4 | 0.372 | 25.5 | 25.6 | 0.048 | 25.5 | |
| Qwen3Setting=VP-ICL2026.05 | 54.5 | 20.1 | 87.7 | 58.2 | 0.476 | 52 | 49.4 | 0.528 | 28.2 | |
| Qwen3Setting=ICL2026.05 | 53.5 | 18.6 | 87.3 | 55.9 | 0.692 | 33.5 | 34.4 | 0.179 | 26.5 | |
| Llama3.2Setting=SFT-ind.2026.05 | 51.2 | 27.1 | 88.7 | 54.9 | 0.113 | 87.3 | 90.9 | 0.946 | 28.2 | |
| Llama3.2Setting=CAPO-strict2026.05 | 51.2 | 26.2 | 88.7 | 54 | 0.121 | 88.5 | 92.4 | 0.964 | 29.7 | |
| Qwen3Setting=VP2026.05 | 49.2 | 13.1 | 85.7 | 51.7 | 4.391 | 21.7 | 23.4 | -0.348 | 24 | |
| Qwen3Setting=Base2026.05 | 47.2 | 15.9 | 86.6 | 51.6 | 2.394 | 24 | 23.9 | -0.034 | 23.7 | |
| Llama3.2Setting=VP-ICL2026.05 | 41.2 | 21 | 87.7 | 55.2 | 0.539 | 26 | 26 | 0.432 | 27.5 | |
| Llama3.2Setting=ICL2026.05 | 38 | 21.4 | 87.6 | 55 | 0.476 | 29.1 | 29.5 | 0.375 | 26 | |
| Llama3.2Setting=VP2026.05 | 33.8 | 14.8 | 86.3 | 46.7 | 1.907 | 37.9 | 35.8 | 0.325 | 25.3 | |
| Llama3.2Setting=Base2026.05 | 31.5 | 15.3 | 86.5 | 48.3 | 2.036 | 26.5 | 25.8 | 0.023 | 26.5 |