# The Need for Speed: Pruning Transformers with One Recipe

Source: [The Need for Speed: Pruning Transformers with One Recipe](<https://www.sota2.com/research/paper/68fa72b41e209131cea58097>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:32:05.519Z

Authors: Samir Khaki, Konstantinos N. Plataniotis

Published: 2024-03-26T17:55:58.000Z

## Abstract

We introduce the $\\textbf{O}$ne-shot $\\textbf{P}$runing $\\textbf{T}$echnique for $\\textbf{I}$nterchangeable $\\textbf{N}$etworks ($\\textbf{OPTIN}$) framework as a tool to increase the efficiency of pre-trained transformer architectures $\\textit{without requiring re-training}$. Recent works have explored improving transformer efficiency, however often incur computationally expensive re-training procedures or depend on architecture-specific characteristics, thus impeding practical wide-scale adoption. To address these shortcomings, the OPTIN framework leverages intermediate feature distillation, capturing the long-range dependencies of model parameters (coined $\\textit{trajectory}$), to produce state-of-the-art results on natural language, image classification, transfer learning, and semantic segmentation tasks $\\textit{without re-training}$. Given a FLOP constraint, the OPTIN framework will compress the network while maintaining competitive accuracy performance and improved throughput. Particularly, we show a $\\leq 2$% accuracy degradation from NLP baselines and a $0.5$% improvement from state-of-the-art methods on image classification at competitive FLOPs reductions. We further demonstrate the generalization of tasks and architecture with comparative performance using Mask2Former for semantic segmentation and cnn-style networks. OPTIN presents one of the first one-shot efficient frameworks for compressing transformer architectures that generalizes well across different class domains, in particular: natural language and image-related tasks, without $\\textit{re-training}$.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.17921>)

[PDF](<https://arxiv.org/pdf/2403.17921.pdf>)