# Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning

Source: [Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning](<https://www.sota2.com/research/paper/68fa691c1e209131cea20510>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:53:51.580Z

Authors: Zeming Li, Songtao Liu, Jian Sun

Published: 2021-01-19T09:27:03.000Z

## Abstract

In this paper, we present a novel approach, Momentum$^2$ Teacher, for student-teacher based self-supervised learning. The approach performs momentum update on both network weights and batch normalization (BN) statistics. The teacher's weight is a momentum update of the student, and the teacher's BN statistics is a momentum update of those in history. The Momentum$^2$ Teacher is simple and efficient. It can achieve the state of the art results (74.5\\%) under ImageNet linear evaluation protocol using small-batch size(\\eg, 128), without requiring large-batch training on special hardware like TPU or inefficient across GPU operation (\\eg, shuffling BN, synced BN). Our implementation and pre-trained models will be given on GitHub\\footnote{https://github.com/zengarden/momentum2-teacher}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2101.07525>)

[PDF](<https://arxiv.org/pdf/2101.07525.pdf>)