# Composing Linear Layers from Irreducibles

Source: [Composing Linear Layers from Irreducibles](<https://www.sota2.com/research/paper/68fa7a1c1e209131cea81406>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-12T05:00:05.418Z

Authors: Travis Pence, Daisuke Yamada, Vikas Singh

Published: 2025-07-15T19:46:00.000Z

## Abstract

Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. We investigate this compositional structure in linear layers by asking: can we identify/synthesize linear transformations from a minimal set of geometric primitives? Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors -- geometric objects encoding oriented planes -- and introduce a differentiable algorithm that decomposes them into products of rotors. This construction uses only O(log^2 d) parameters, versus O(d^2) required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on C4 (Log-PPL)](<https://www.sota2.com/research/sota/language-modeling-on-c4-log-ppl.md>): Language Modeling; C4; Log-PPL: 2.854.

- [Common Sense Reasoning on Hellaswag](<https://www.sota2.com/research/sota/common-sense-reasoning-on-hellaswag.md>): Common Sense Reasoning; HellaSwag; Accuracy: 50.2.

- [Question Answering on ARC Challenge](<https://www.sota2.com/research/sota/question-answering-on-arc-challenge.md>): Question Answering; ARC Challenge; Accuracy: 61.34.

- [Language Modeling on WikiText-2](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-2.md>): Language Modeling; WikiText-2; Perplexity (PPL): not recorded.

- [Language Modeling on PTB](<https://www.sota2.com/research/sota/language-modeling-on-ptb.md>): Language Modeling; PTB; Perplexity: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.11688>)

[PDF](<https://arxiv.org/pdf/2507.11688.pdf>)