# Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

Source: [Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks](<https://www.sota2.com/research/paper/69ddca057d0172140ec79947>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-15T05:00:04.591Z

Authors: Wang Zixian

Published: 2026-04-08T02:04:47.000Z

## Abstract

Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and increased redundancy relative to the main branch. We study a complementary design principle in which auxiliary quadratic features contribute only information not already captured by the dominant hidden representation. To this end, we propose Orthogonal Quadratic Complements (OQC), which construct a low-rank quadratic auxiliary branch and explicitly project it onto the orthogonal complement of the main branch before injection. We further study an efficient low-rank realization (OQC-LR) and gated extensions (OQC-static and OQC-dynamic).   Under a parameter-matched Deep-ViT and CIFAR-100 protocol with a fixed penultimate residual readout, full OQC improves an AFBO baseline from 64.25 +/- 0.22 to 65.59 +/- 0.22, while OQC-LR reaches 65.52 +/- 0.25 with a substantially better speed-accuracy tradeoff. On TinyImageNet, the gated extension OQC-dynamic achieves 51.88 +/- 0.32, improving the baseline (50.45 +/- 0.21) by 1.43 points and outperforming all ungated variants. Mechanism analyses show near-zero post-projection auxiliary-main overlap together with improved representation geometry and class separation. The full family, including both ungated and gated variants, generalizes consistently across both datasets.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on TinyImageNet cross-dataset transfer](<https://www.sota2.com/research/sota/image-classification-on-tinyimagenet-cross-dataset-transfer.md>): Image Classification; TinyImageNet cross-dataset transfer; Accuracy: 51.88.

- [Image Classification on CIFAR-100 (test) (Standard Metrics)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test-standard-metrics.md>): Image Classification; CIFAR-100 (test); Top-1 Accuracy: 65.59.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.09709>)

[PDF](<https://arxiv.org/pdf/2604.09709.pdf>)