# DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training

Source: [DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training](<https://www.sota2.com/research/paper/68fa6ea91e209131cea412cd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:02:50.787Z

Authors: Joya Chen, Kai Xu, Yuhui Wang, Yifei Cheng, Angela Yao

Published: 2022-02-28T14:12:00.000Z

## Abstract

A standard hardware bottleneck when training deep neural networks is GPU memory. The bulk of memory is occupied by caching intermediate tensors for gradient computation in the backward pass. We propose a novel method to reduce this footprint - Dropping Intermediate Tensors (DropIT). DropIT drops min-k elements of the intermediate tensors and approximates gradients from the sparsified tensors in the backward pass. Theoretically, DropIT reduces noise on estimated gradients and therefore has a higher rate of convergence than vanilla-SGD. Experiments show that we can drop up to 90\\% of the intermediate tensor elements in fully-connected and convolutional layers while achieving higher testing accuracy for Visual Transformers and Convolutional Neural Networks on various tasks (e.g., classification, object detection, instance segmentation). Our code and models are available at https://github.com/chenjoya/dropit.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2202.13808>)

[PDF](<https://arxiv.org/pdf/2202.13808.pdf>)
