Annotator agreement on SPS (test)
57.74Krippendorff's Alphaslm-judge
Evaluation Results
| Method | Links | |
|---|---|---|
| slm-judgeBackbone=Qwen3-1.7B, PEFT=4-bit quantized, Data Augmentation=true2026.04 | 57.74 | |
| Early stopping without augmentationBackbone=Qwen3-1.7B, Data Augmentation=false, Strategy=Early stopping2026.04 | 43.8 | |
| Full finetuning without augmentationBackbone=Qwen3-1.7B, Data Augmentation=false2026.04 | 43.04 | |
| LoRA dropoutBackbone=Qwen3-1.7B, p=0.12026.04 | 40.67 | |
| GPT-5-mini-2025-08-07Prompting Strategy=Zero-shot2026.04 | 24.62 | |
| GPT-5.2-chatPrompting Strategy=Zero-shot2026.04 | 20.54 | |
| GPT-4oPrompting Strategy=Zero-shot2026.04 | 19.64 | |
| GPT-5-nanoPrompting Strategy=Zero-shot2026.04 | 10.65 | |
| GPT-5.2-chatPrompting Strategy=Few-shot, Optimizer=MIPROv22026.04 | 4.71 | |
| GPT-4oPrompting Strategy=Few-shot, Optimizer=MIPROv22026.04 | 1.01 |