# Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Source: [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](<https://www.sota2.com/research/paper/68fa65041e209131cea07f12>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T05:56:48.640Z

Authors: Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson

Published: 2018-02-27T17:13:28.000Z

## Abstract

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mode Connectivity on CIFAR10](<https://www.sota2.com/research/sota/mode-connectivity-on-cifar10.md>): Mode Connectivity; CIFAR10; Worst Case Training Loss: 0.022.

- [Image Classification on CIFAR-10 (test) (Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-accuracy.md>): Image Classification; CIFAR-10 (test); Accuracy: 96.36.

- [Image Classification on CIFAR-100 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test.md>): Image Classification; CIFAR-100 (test); Accuracy: not recorded.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: not recorded.

- [Image Classification on Tiny-ImageNet (test)](<https://www.sota2.com/research/sota/image-classification-on-tiny-imagenet-test.md>): Image Classification; Tiny ImageNet (test); Accuracy: not recorded.

- [Image Classification on ImageNet (test)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-test.md>): Image Classification; ImageNet (test); Top-1 Acc: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1802.10026>)

[PDF](<https://arxiv.org/pdf/1802.10026.pdf>)