# Tensor Cache: Eviction-conditioned Associative Memory for Transformers

Source: [Tensor Cache: Eviction-conditioned Associative Memory for Transformers](<https://www.sota2.com/research/paper/6a13d7816f71c5ae7dc9e814>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-26T05:00:05.667Z

Authors: Kabir Swain, Sijie Han, Daniel Karl I. Weidele, Mauro Martino, Antonio Torralba

Published: 2026-05-21T00:21:51.000Z

## Abstract

Autoregressive Transformer KV caches grow linearly with context length; sliding-window caching bounds memory but discards evicted tokens entirely, so relevant evidence outside the window becomes inaccessible. We introduce \\emph{Tensor Cache}, a two-level cache that pairs sliding-window softmax attention as a first-level cache (L1) with a fixed-size outer-product fast-weight memory as a second-level cache (L2) fed by KV pairs evicted from the window. Recent tokens remain in exact local attention; evicted pairs are compressed into a per-layer matrix $A$ and read by future queries through a single matrix multiplication, exploiting the linear-attention identity $q\_t(k\_i \\otimes v\_i)=\\langle q\_t,k\_i\\rangle v\_i$. A learned scalar gate fuses the L1 and L2 outputs, and per-head decay and write-rate parameters are trained end-to-end. The outer-product memory and the read identity are well-known; our contribution is their use as an L2 cache fed exclusively by sliding-window evictions, plus identifying that the common chunked-mean training shortcut $A\\!\\leftarrow\\!\\lambda A\\!+\\!\\eta(\\bar k\\!\\otimes\\!\\bar v)$ silently introduces $C^2{-}C$ spurious cross-token outer products per chunk, and closing the gap with a parallel weighted-sum scan equivalent to per-token writes within float32 epsilon. Across systems scaling, controlled associative recall, long-context language modeling, and memory-capacity diagnostics, Tensor Cache improves the memory--quality frontier over bounded-state baselines.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on Shakespeare 1M params (val)](<https://www.sota2.com/research/sota/language-modeling-on-shakespeare-1m-params-val.md>): Language Modeling; Shakespeare 1M params (val); NLL: 4.79.

- [Long-context language modeling on OpenWebText (val)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-openwebtext-val.md>): Long-context language modeling; OpenWebText (val); NLL (Context=1024): 3.93.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.22884>)

[PDF](<https://arxiv.org/pdf/2605.22884.pdf>)