DSL Generation on RED-Commerce 095
86.94TB ScoreREDPARROT-MAN
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| REDPARROT-MANEncoding=Manually labeled skeletons (Qwen3-embedding-0.6B), LLM=Qwen2.5-72b2026.03 | 86.94 | 79.28 | 70.27 | 96.85 | 51.13 | 10.43 | |
| REDPARROTEncoding=Entity-agnostic embedding model (Qwen3-embedding-0.6B backbone), LLM=Qwen2.5-72b2026.03 | 85.36 | 76.13 | 69.37 | 91.22 | 48.65 | 7.16 | |
| REDPARROT-RAWEncoding=Raw user queries (Qwen3-embedding-0.6B), LLM=Qwen2.5-72b2026.03 | 77.93 | 72.52 | 66.89 | 83.56 | 44.37 | 7.16 | |
| LONGCHAIN-DSReasoning Model=DeepSeek-V32026.03 | 69.66 | 82.7 | 53.93 | 97.53 | 45.62 | 22 | |
| LONGCHAIN-QWENReasoning Model=Qwen2.5-72b, Configuration Model=Qwen2.5-72b2026.03 | 15.05 | 66.42 | 47.19 | 92.81 | 9.66 | 21 |