# Negative Inner-Loop Learning Rates Learn Universal Features

Source: [Negative Inner-Loop Learning Rates Learn Universal Features](<https://www.sota2.com/research/paper/68fa6c141e209131cea31fdb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:03:49.010Z

Authors: Tom Starshak

Published: 2022-03-18T22:43:16.000Z

## Abstract

Model Agnostic Meta-Learning (MAML) consists of two optimization loops: the outer loop learns a meta-initialization of model parameters that is shared across tasks, and the inner loop task-specific adaptation step. A variant of MAML, Meta-SGD, uses the same two loop structure, but also learns the learning-rate for the adaptation step. Little attention has been paid to how the learned learning-rate of Meta-SGD affects feature reuse. In this paper, we study the effect that a learned learning-rate has on the per-task feature representations in Meta-SGD. The learned learning-rate of Meta-SGD often contains negative values. During the adaptation phase, these negative learning rates push features away from task-specific features and towards task-agnostic features.   We performed several experiments on the Mini-Imagenet dataset. Two neural networks were trained, one with MAML, and one with Meta-SGD. The feature quality for both models was tested as follows: strip away the linear classification layer, pass labeled and unlabeled samples through this encoder, classify the unlabeled samples according to their nearest neighbor. This process was performed: 1) after training and using the meta-initialization parameters; 2) after adaptation, and validated on that task; and 3) after adaptation, and validated on a different task. The MAML trained model improved on the task it was adapted to, but had worse performance on other tasks. The Meta-SGD trained model was the opposite; it had worse performance on the task it was adapted to, but improved on other tasks. This confirms the hypothesis that Meta-SGD's negative learning rates cause the model to learn task-agnostic features rather than simply adapt to task specific features.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2203.10185>)

[PDF](<https://arxiv.org/pdf/2203.10185.pdf>)
