# Next-Embedding Prediction Makes Strong Vision Learners

Source: [Next-Embedding Prediction Makes Strong Vision Learners](<https://www.sota2.com/research/paper/6944dc2b423ea42265729d20>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-25T05:00:04.588Z

Authors: Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, Weiyang Jin, Joyce Chai, Saining Xie, Stella X. Yu

Published: 2025-12-18T18:59:58.000Z

## Abstract

Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (Top-1 and Top-5 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-top-1-and-top-5-accuracy.md>): Image Classification; ImageNet; Top-1 Acc: 85.3.

- [CTG Classification on CTG Dataset Antepartum (T1-T4)](<https://www.sota2.com/research/sota/ctg-classification-on-ctg-dataset-antepartum-t1-t4.md>): CTG Classification; CTG Dataset Antepartum (T1-T4); AUC (T1): 0.626.

- [CTG Classification on CTG Dataset Intrapartum (T5-T7)](<https://www.sota2.com/research/sota/ctg-classification-on-ctg-dataset-intrapartum-t5-t7.md>): CTG Classification; CTG Dataset Intrapartum (T5-T7); T5 AUC: 72.5.

- [Semantic Segmentation on ADE20K](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k.md>): Semantic segmentation; ADE20K; mIoU: 54.

- [Image Classification on ImageNet-1K (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-top-1-accuracy.md>): Image Classification; ImageNet-1K; Top-1 Acc: 85.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2512.16922>)

[PDF](<https://arxiv.org/pdf/2512.16922.pdf>)

[Implementation](<https://github.com/SihanXU/nepa>)
