# The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

Source: [The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding](<https://www.sota2.com/research/paper/694a22519eebe296e72aab5c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-03T05:00:04.568Z

Authors: Weichen Fan, Haiwen Diao, Quan Wang, Dahua Lin, Ziwei Liu

Published: 2025-12-22T18:59:57.000Z

## Abstract

Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments demonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art performance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanilla JIT baseline. Our code is avaliable at: https://github.com/WeichenFan/UAE.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Reconstruction on ImageNet-1K (full set)](<https://www.sota2.com/research/sota/image-reconstruction-on-imagenet-1k-full-set.md>): Image Reconstruction; ImageNet-1K (full set); PSNR: 33.08.

- [Image Reconstruction on MS-COCO 2017 (full set)](<https://www.sota2.com/research/sota/image-reconstruction-on-ms-coco-2017-full-set.md>): Image Reconstruction; MS-COCO 2017 (full set); PSNR: 32.84.

- [Class-Conditional Image Generation on ImageNet 256x256 (train val)](<https://www.sota2.com/research/sota/class-conditional-image-generation-on-imagenet-256x256-train-val.md>): Class-conditional Image Generation; ImageNet 256x256 (train val); FID: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2512.19693>)

[PDF](<https://arxiv.org/pdf/2512.19693.pdf>)

[Implementation](<https://github.com/WeichenFan/UAE>)