# Efficiency for Free: Ideal Data Are Transportable Representations

Source: [Efficiency for Free: Ideal Data Are Transportable Representations](<https://www.sota2.com/research/paper/68fa75901e209131cea68a7a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:36:51.485Z

Authors: Peng Sun, Yi Jiang, Tao Lin

Published: 2024-05-23T15:06:02.000Z

## Abstract

Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution. In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives. Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, task- and architecture-agnostic model (referred to as the \`prior model' in this paper) can effectively produce efficient data. Building on this insight, we propose the Representation Learning Accelerator (\\algopt), which promotes the formation and utilization of efficient data, thereby accelerating representation learning. Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by 50% while maintaining the same accuracy as the model trained with the original BYOL, which requires 100% cost. Our code is available at: \\url{https://github.com/LINs-lab/ReLA}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.14669>)

[PDF](<https://arxiv.org/pdf/2405.14669.pdf>)