Intent Understanding on Highlight-Intent
63.97AccuracyIntPro
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| IntProModel=Qwen3-4B2026.02 | 63.97 | 48.1 | 61.6 | |
| IntProModel=Qwen2.5-3B2026.02 | 63.36 | 48.64 | 60.64 | |
| IntProModel=Llama3.2-3B2026.02 | 59.24 | 44.24 | 56.71 | |
| GPT-4oModel=GPT-4o2026.02 | 48.55 | 30.02 | 46.2 | |
| IntProBackbone=Qwen3-4B, Training strategy=IntPro (Ours)2026.02 | 48.28 | 30.56 | 43.14 | |
| Naive GRPOBackbone=Qwen3-4B, Training strategy=Naive GRPO2026.02 | 47.17 | 28.12 | 41.28 | |
| IntProBackbone=Qwen2.5-3B, Training strategy=IntPro (Ours)2026.02 | 46.34 | 28.75 | 40.82 | |
| Qwen3-30B-A3BModel=Qwen3-30B-A3B2026.02 | 46.23 | 28.15 | 43.87 | |
| SFT with Answer-onlyBackbone=Qwen3-4B, Training strategy=SFT with Answer-only2026.02 | 46.12 | 28.45 | 40.56 | |
| IntProBackbone=Llama3.2-3B, Training strategy=IntPro (Ours)2026.02 | 45.24 | 30.52 | 41.35 | |
| SFT with Answer-onlyBackbone=Qwen2.5-3B, Training strategy=SFT with Answer-only2026.02 | 44.42 | 23.55 | 37.11 | |
| SFT with Retrieval-conditionedBackbone=Qwen3-4B, Training strategy=SFT with Retrieval-conditioned2026.02 | 44.25 | 27.14 | 38.86 | |
| Naive GRPOBackbone=Qwen2.5-3B, Training strategy=Naive GRPO2026.02 | 43.82 | 26.14 | 37.65 | |
| RoBERTa-baseBackbone=RoBERTa-base2026.02 | 43.21 | 22.37 | 35.8 | |
| Naive GRPOBackbone=Llama3.2-3B, Training strategy=Naive GRPO2026.02 | 42.75 | 26.84 | 38.92 | |
| SFT with Retrieval-conditionedBackbone=Qwen2.5-3B, Training strategy=SFT with Retrieval-conditioned2026.02 | 41.98 | 22.67 | 35.24 | |
| BERT-baseBackbone=BERT-base2026.02 | 38.17 | 13.82 | 29.66 | |
| GPT-4o (w/ retrieval)Backbone=GPT-4o, Retrieval usage=true2026.02 | 36.97 | 17.82 | 32.45 | |
| SFT with Answer-onlyBackbone=Llama3.2-3B, Training strategy=SFT with Answer-only2026.02 | 36.18 | 24.88 | 35.58 | |
| Qwen3-30B-A3B (w/ retrieval)Backbone=Qwen3-30B-A3B, Retrieval usage=true2026.02 | 34.56 | 15.67 | 29.82 | |
| SFT with Retrieval-conditionedBackbone=Llama3.2-3B, Training strategy=SFT with Retrieval-conditioned2026.02 | 34.25 | 23.94 | 33.72 | |
| GPT-4oBackbone=GPT-4o2026.02 | 30.62 | 13.71 | 26.88 | |
| Qwen3-30B-A3BBackbone=Qwen3-30B-A3B2026.02 | 28.45 | 12.39 | 24.67 |