Job-skill Extraction on SKILLSPAN
75.22PrecisionSRICL-Qwen2.5-14B (Our method)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SRICL-Qwen2.5-14B (Our method)Evaluation Protocol=RELAX / NER-STYLE, Model=Qwen2.5-14B2026.04 | 75.22 | 77.43 | 76.31 | |
| 5-shotEvaluation Protocol=RELAX / NER-STYLE, Setting=5-shot2026.04 | 74.47 | 61.95 | 67.63 | |
| +kNNEvaluation Protocol=RELAX / NER-STYLE, Setting=5-shot, Retrieval (kNN)=true2026.04 | 71.67 | 61.58 | 66.25 | |
| SRICL-Qwen2.5-14B (Our method)Evaluation Protocol=STRICT / NER-STYLE, Model=Qwen2.5-14B2026.04 | 53.83 | 55.36 | 54.59 | |
| 5-shotEvaluation Protocol=STRICT / NER-STYLE, Setting=5-shot2026.04 | 35.46 | 29.5 | 32.21 | |
| Qwen2.5 (5-shot + kNN + data-specific prompt)Evaluation Protocol=RELAX / NER-STYLE, Model=Qwen2.5, Setting=5-shot, Retrieval (kNN)=true, Prompting Strategy=data-specific prompt2026.04 | 33.14 | 31.83 | 32.48 | |
| Llama-3-8B-SRICL(B8-Our Method)Evaluation Protocol=STRICT / NER-STYLE, Model=Llama-3-8B, Method Variant=B82026.04 | 31.5 | 55.09 | 40.08 | |
| +kNNEvaluation Protocol=STRICT / NER-STYLE, Setting=5-shot, Retrieval (kNN)=true2026.04 | 26.96 | 23.17 | 24.92 | |
| +kNN+specificEvaluation Protocol=RELAX / NER-STYLE, Setting=5-shot, Retrieval (kNN)=true, Prompting Strategy=specific2026.04 | 26.77 | 60.18 | 37.06 | |
| zero-shot+specific (GPT-3.5)Evaluation Protocol=RELAX / NER-STYLE, Prompting Strategy=specific, Model=GPT-3.5, Setting=zero-shot2026.04 | 21.43 | 9.68 | 13.33 | |
| +kNN+specificEvaluation Protocol=STRICT / NER-STYLE, Setting=5-shot, Retrieval (kNN)=true, Prompting Strategy=specific2026.04 | 10.89 | 24.48 | 15.08 | |
| Qwen2.5 (5-shot + kNN + data-specific prompt)Evaluation Protocol=STRICT / NER-STYLE, Model=Qwen2.5, Setting=5-shot, Retrieval (kNN)=true, Prompting Strategy=data-specific prompt2026.04 | 9.26 | 8.89 | 9.07 | |
| Llama-3-8B (SRICL- Our Method)Evaluation Protocol=RELAX / NER-STYLE, Model=Llama-3-8B2026.04 | 9.26 | 8.89 | 9.07 | |
| GPT-3.5 Version (+kNN+specific)Evaluation Protocol=STRICT / NER-STYLE, Retrieval (kNN)=true, Prompting Strategy=specific, Model=GPT-3.52026.04 | 9.1 | 26.6 | 13.7 | |
| zero-shot+specific (GPT-3.5)Evaluation Protocol=STRICT / NER-STYLE, Prompting Strategy=specific, Model=GPT-3.5, Setting=zero-shot2026.04 | 3.08 | 11.94 | 4.9 |