# Composing Ensembles of Pre-trained Models via Iterative Consensus Source: [Composing Ensembles of Pre-trained Models via Iterative Consensus]() Collection index: [llms.txt]() Data updated: 2026-01-04T03:45:21.883Z Authors: Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch Published: 2022-10-20T18:46:31.000Z ## Abstract Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions. In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths of each expert model. Results show that the proposed method can be used as a general purpose framework for a wide range of zero-shot multimodal tasks, such as image generation, video question answering, mathematical reasoning, and robotic manipulation. Project page: https://energy-based-model.github.io/composing-pretrained-models. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Generation on ImageNet (test)](): Image Generation; ImageNet (test); Inception Score: 34.952. - [Robot Manipulation on Ravens 2 Relations, Text Goal](): Robot Manipulation; Ravens 2 Relations, Text Goal; Success Rate: 67.5. - [Robot Manipulation on Ravens 2 Relations, Image Goal](): Robot Manipulation; Ravens 2 Relations, Image Goal; Success Rate: 52.6. - [Robot Manipulation on Ravens 3 Relations, Text Goal](): Robot Manipulation; Ravens 3 Relations, Text Goal; Success Rate: 75. - [Robot Manipulation on Ravens 3 Relations, Image Goal](): Robot Manipulation; Ravens 3 Relations, Image Goal; Success Rate: 65.3. - [Mathematical Reasoning on GSM8K (test)](): Mathematical Reasoning; GSM8K (test); Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()