# Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian

Source: [Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian](<https://www.sota2.com/research/paper/6a3a1326206542251d7c70e3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-02T05:00:05.660Z

Authors: Rome Thorstenson

Published: 2026-06-20T04:35:51.000Z

## Abstract

The Categorical Jacobian of Zhang et al. (2024) reads protein contacts from a language model by perturbing every residue with every alternative amino acid, about $19L$ forward passes. We show the signal it reconstructs is already concentrated in a small subset of attention heads: averaging the top-$K$ contact-relevant heads -- selected on as few as 10 labeled proteins, with no fitted per-pair or per-head weights -- recovers contacts in a single forward pass and matches or beats the Categorical Jacobian for every bidirectional model where it is defined (bar the smallest, 8M). Our primary test is leakage-clean: on a CAMEO split where neither selection nor evaluation touches data the models have plausibly memorized, the head readout beats the Categorical Jacobian on ESM-2-650M by +9pp ($N = 29$, $p \< 0.001$), with the within-model margin reproducing across architectures. Ablations localize the gain to labeled head selection, not to averaging: at a matched label budget the unweighted mean ties a supervised $L\_1$ logistic regression on the same heads. Both methods fall 30-36pp from their in-distribution Zhang numbers to the leakage-clean split, which we read as an upper bound on how much prior numbers reflect pretraining overlap. We additionally introduce representation-CJ, a hidden-state generalization of the Jacobian to architectures without a masked-LM head (the output-head-independent analogue of logit-CJ), agreeing with the Categorical Jacobian where both are defined (per-protein Pearson $r \\approx 0.95$); show that the optimal $K$ tracks how diffusely a model spreads its contact heads; and find both methods lose the signal on the two causal LMs we test, suggesting attention-encoded pair structure may depend on bidirectional pretraining.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Contact Prediction on CAMEO-PTA25 L ≥ 75, N=29 2022-09 cutoff](<https://www.sota2.com/research/sota/contact-prediction-on-cameo-pta25-l-75-n-29-2022-09-cutoff.md>): Contact Prediction; CAMEO-PTA25 L ≥ 75, N=29 2022-09 cutoff; Precision @ L/2 (long): 46.3.

- [Protein contact prediction on CAMEO PTA25 (L ≥ 75, N=29)](<https://www.sota2.com/research/sota/protein-contact-prediction-on-cameo-pta25-l-75-n-29.md>): Protein Contact Prediction; CAMEO PTA25 (L ≥ 75, N=29); Precision @ L/2 (long): 45.9.

- [Protein Contact Prediction on Zhang 50](<https://www.sota2.com/research/sota/protein-contact-prediction-on-zhang-50.md>): Protein Contact Prediction; Zhang 50; repr-CJ: 0.79.

- [Protein Contact Prediction on Zhang eval-200](<https://www.sota2.com/research/sota/protein-contact-prediction-on-zhang-eval-200.md>): Protein Contact Prediction; Zhang eval-200; Top-1 Head: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.21876>)

[PDF](<https://arxiv.org/pdf/2606.21876.pdf>)
