Annotator Simulation on VariErr
63.8Label AccuracySFT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SFTBackbone=Qwen32026.05 | 63.8 | 28.3 | 0.084 | 84.5 | 0.859 | 30 | |
| CAPOBackbone=Qwen32026.05 | 62.7 | 28.1 | 0.081 | 86.7 | 0.888 | 32.8 | |
| VP-ICLBackbone=Qwen32026.05 | 54.5 | 20.1 | 0.476 | 49.4 | 0.528 | 28.2 | |
| ICLBackbone=Qwen32026.05 | 53.5 | 18.6 | 0.692 | 34.4 | 0.179 | 26.5 | |
| SFTBackbone=Llama3.22026.05 | 51.2 | 27.1 | 0.113 | 90.9 | 0.946 | 28.2 | |
| CAPOBackbone=Llama3.22026.05 | 51.2 | 26.2 | 0.121 | 92.4 | 0.964 | 29.7 | |
| VPBackbone=Qwen32026.05 | 49.2 | 13.1 | 4.391 | 23.4 | -0.348 | 24 | |
| BaseBackbone=Qwen32026.05 | 47.2 | 15.9 | 2.394 | 23.9 | -0.034 | 23.7 | |
| VP-ICLBackbone=Llama3.22026.05 | 41.2 | 21 | 0.539 | 26 | 0.432 | 27.5 | |
| ICLBackbone=Llama3.22026.05 | 38 | 21.4 | 0.476 | 29.5 | 0.375 | 26 | |
| VPBackbone=Llama3.22026.05 | 33.8 | 14.8 | 1.907 | 35.8 | 0.325 | 25.3 | |
| BaseBackbone=Llama3.22026.05 | 31.5 | 15.3 | 2.036 | 25.8 | 0.023 | 26.5 |