# EnsembleNet: End-to-End Optimization of Multi-headed Models

Source: [EnsembleNet: End-to-End Optimization of Multi-headed Models](<https://www.sota2.com/research/paper/68fa66781e209131cea105b3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:50:28.993Z

Authors: Hanhan Li, Joe Yue-Hei Ng, Paul Natsev

Published: 2019-05-24T00:31:11.000Z

## Abstract

Ensembling is a universally useful approach to boost the performance of machine learning models. However, individual models in an ensemble were traditionally trained independently in separate stages without information access about the overall ensemble. Many co-distillation approaches were proposed in order to treat model ensembling as first-class citizens. In this paper, we reveal a deeper connection between ensembling and distillation, and come up with a simpler yet more effective co-distillation architecture. On large-scale datasets including ImageNet, YouTube-8M, and Kinetics, we demonstrate a general procedure that can convert a single deep neural network to a multi-headed model that has not only a smaller size but also better performance. The model can be optimized end-to-end with our proposed co-distillation loss in a single stage without human intervention.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1905.09979>)

[PDF](<https://arxiv.org/pdf/1905.09979.pdf>)