# Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective Source: [Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:00:33.644Z Authors: Qishuai Wen, Chun-Guang Li Published: 2024-11-05T12:10:02.000Z ## Abstract State-of-the-art methods for Transformer-based semantic segmentation typically adopt Transformer decoders that are used to extract additional embeddings from image embeddings via cross-attention, refine either or both types of embeddings via self-attention, and project image embeddings onto the additional embeddings via dot-product. Despite their remarkable success, these empirical designs still lack theoretical justifications or interpretations, thus hindering potentially principled improvements. In this paper, we argue that there are fundamental connections between semantic segmentation and compression, especially between the Transformer decoders and Principal Component Analysis (PCA). From such a perspective, we derive a white-box, fully attentional DEcoder for PrIncipled semantiC segemenTation (DEPICT), with the interpretations as follows: 1) the self-attention operator refines image embeddings to construct an ideal principal subspace that aligns with the supervision and retains most information; 2) the cross-attention operator seeks to find a low-rank approximation of the refined image embeddings, which is expected to be a set of orthonormal bases of the principal subspace and corresponds to the predefined classes; 3) the dot-product operation yields compact representation for image embeddings as segmentation masks. Experiments conducted on dataset ADE20K find that DEPICT consistently outperforms its black-box counterpart, Segmenter, and it is light weight and more robust. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Semantic Segmentation on ADE20K (val)](): Semantic segmentation; ADE20K (val); mIoU: not recorded. - [Semantic Segmentation on Cityscapes (val)](): Semantic segmentation; Cityscapes (val); mIoU: not recorded. - [Semantic Segmentation on PASCAL Context (val)](): Semantic segmentation; PASCAL Context (val); mIoU: not recorded. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()