Compositional Reasoning on Harder-set
51.3Strict AccuracyQwen2.5-7B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen2.5-7BParameters=7B2026.05 | 51.3 | 47.6 | 50.4 | 50.4 | |
| Mistral-7BParameters=7B2026.05 | 47.9 | 100 | 47.9 | 47.9 | |
| DeepHermes-3-8BParameters=8B2026.05 | 46.7 | 71.4 | 46.7 | 40.2 | |
| Qwen3-8BParameters=8B2026.05 | 3.4 | 14.3 | 2.8 | 3.4 | |
| Llama2-7B-ChatParameters=7B2026.05 | 0 | — | — | 0 | |
| Llama2-13B-ChatParameters=13B2026.05 | 0 | 76 | 76 | 0 |