Compositional Reasoning on WebAggregatorQA
70.8Level-1 ScoreWebAggregator
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen2.5-32B, Evaluation Protocol=Pass@32025.10 | 70.8 | 22.2 | 19.4 | 28.9 | — | — | |
| Claude-3.7-sonnetSetting=Zero-shot Foundations2025.10 | 66.7 | 25.3 | 11.1 | 28.3 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen3-32B, Evaluation Protocol=Pass@32025.10 | 66.7 | 35.4 | 13.9 | 35.2 | — | — | |
| GPT-4.1Setting=Zero-shot Foundations2025.10 | 62.4 | 22.2 | 11.1 | 25.8 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen3-8B, Evaluation Protocol=Pass@32025.10 | 62.4 | 21.2 | 11.1 | 25.2 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen3-32B, Evaluation Protocol=Pass@12025.10 | 62.4 | 24.2 | 8.3 | 26.4 | — | — | |
| GPT-5Setting=Zero-shot Foundations2025.10 | 58.3 | 24.5 | 22.2 | 28.9 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen2.5-7B, Evaluation Protocol=Pass@32025.10 | 54.2 | 22.2 | 19.4 | 26.4 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen2.5-32B, Evaluation Protocol=Pass@12025.10 | 54.2 | 15.2 | 11.1 | 20.1 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen3-8B, Evaluation Protocol=Pass@12025.10 | 54.2 | 11.1 | 5.6 | 16.4 | — | — | |
| Qwen3-32BSetting=Zero-shot Foundations2025.10 | 45.8 | 10.1 | 5.6 | 14.5 | — | — | |
| WebAggregatorSetting=Fine-tuned Foundations, Backbone=Qwen2.5-7B, Evaluation Protocol=Pass@12025.10 | 37.5 | 11.1 | 8.3 | 14.5 | — | — | |
| Qwen3-8BSetting=Zero-shot Foundations2025.10 | 30.8 | 5.1 | 5.6 | 9.4 | — | — | |
| Qwen2.5-7BSetting=Zero-shot Foundations2025.10 | 27.3 | 3.4 | 2.8 | 6.3 | — | — | |
| Qwen2.5-32BSetting=Zero-shot Foundations2025.10 | 25 | 10.1 | 5.6 | 11.3 | — | — | |
| Claude-3.7-sonnetSetting=Non-Agentic2025.10 | 18.5 | 5.1 | 2.8 | 6.8 | — | — | |
| GPT-4.1Setting=Non-Agentic2025.10 | 15.4 | 4 | 2.8 | 5.6 | — | — | |
| Qwen3-32BSetting=Non-Agentic2025.10 | 8.3 | 1 | 0 | 1.9 | — | — | |
| Qwen2.5-7BSetting=Non-Agentic2025.10 | 4.2 | 1 | 0 | 1.3 | — | — | |
| Qwen2.5-32BSetting=Non-Agentic2025.10 | 4.2 | 1 | 0 | 1.3 | — | — | |
| Qwen3-8BSetting=Non-Agentic2025.10 | 4.2 | 1 | 2.8 | 1.9 | — | — |