# Reward Modeling for Multi-Agent Orchestration

Source: [Reward Modeling for Multi-Agent Orchestration](<https://www.sota2.com/research/paper/6a2b92d4d33dfb23e783b2dd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-13T05:00:13.110Z

Authors: King Yeung Tsang, Zihao Zhao, Vishal Venkataramani, Haizhou Shi, Zixuan Ke, Semih Yavuz, Shafiq Joty, Hao Wang

Published: 2026-06-11T17:16:24.000Z

## Abstract

Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mathematical Reasoning on AIME 24&25 (Acc, Maj. Vote)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-24-and-25-acc-maj-vote.md>): Mathematical Reasoning; AIME 24&25; Accuracy: 67.5.

- [Web-based QA on BrowseComp+](<https://www.sota2.com/research/sota/web-based-qa-on-browsecomp-plus.md>): Web-based QA; BrowseComp+; Accuracy: 10.

- [MAS Test-Time Scaling on BrowseComp+](<https://www.sota2.com/research/sota/mas-test-time-scaling-on-browsecomp-plus.md>): MAS Test-Time Scaling; BrowseComp+; Accuracy: 14.

- [Multi-hop Reasoning on HotpotQA (Accuracy, Majority Vote)](<https://www.sota2.com/research/sota/multi-hop-reasoning-on-hotpotqa-accuracy-majority-vote.md>): Multi-hop Reasoning; HotpotQA; Accuracy: 41.63.

- [MAS Test-Time Scaling on AIME 24&25](<https://www.sota2.com/research/sota/mas-test-time-scaling-on-aime-24-and-25.md>): MAS Test-Time Scaling; AIME 24&25; Accuracy: 68.33.

- [Scientific Reasoning on GPQA (Acc., Maj. Vote)](<https://www.sota2.com/research/sota/scientific-reasoning-on-gpqa-acc-maj-vote.md>): Scientific Reasoning; GPQA; Accuracy: 66.7.

- [MAS Test-Time Scaling on HotpotQA](<https://www.sota2.com/research/sota/mas-test-time-scaling-on-hotpotqa.md>): MAS Test-Time Scaling; HotpotQA; Accuracy: 42.5.

- [MAS Test-Time Scaling on GPQA](<https://www.sota2.com/research/sota/mas-test-time-scaling-on-gpqa.md>): MAS Test-Time Scaling; GPQA; Accuracy: 66.16.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.13598>)

[PDF](<https://arxiv.org/pdf/2606.13598.pdf>)
