DSL Generation on RED-Commerce 0916
91.15TBREDPARROT-MAN
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| REDPARROT-MANEncoding=Manually labeled skeletons (Qwen3-embedding-0.6B), LLM=Qwen2.5-72b2026.03 | 91.15 | 84.99 | 72.39 | 93.57 | 58.45 | 10.45 | |
| REDPARROTEncoding=Entity-agnostic embedding model (Qwen3-embedding-0.6B backbone), LLM=Qwen2.5-72b2026.03 | 90.62 | 81.5 | 70.24 | 91.42 | 57.64 | 7.15 | |
| REDPARROT-RAWEncoding=Raw user queries (Qwen3-embedding-0.6B), LLM=Qwen2.5-72b2026.03 | 87.67 | 78.02 | 65.95 | 86.06 | 47.45 | 6.63 | |
| LONGCHAIN-DSReasoning Model=DeepSeek-V32026.03 | 70.25 | 80.97 | 51.48 | 95.98 | 45.84 | 25 | |
| LONGCHAIN-QWENReasoning Model=Qwen2.5-72b, Configuration Model=Qwen2.5-72b2026.03 | 18.5 | 64.76 | 47.19 | 90.62 | 11.53 | 22 |