# Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models

Source: [Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models](<https://www.sota2.com/research/paper/6a45f092dc71372d7795a2de>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-03T05:00:11.862Z

Authors: Miko{\\l}aj S{\\l}owikowski, Maciej Witold Majewski

Published: 2026-07-01T12:18:12.000Z

## Abstract

This work studies the hidden-state inversion problem: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states. Rather than treating inversion as a one-shot reconstruction, we study it as a continuous embedding-space optimisation in which a soft proxy is driven towards the leaked target without any hard-token projection during the search, and a token is committed only once, at the end of the inner loop. This design choice has two consequences which are the main focus of this paper. First, keeping the optimisation entirely in continuous space exposes a rich set of internal signals: rank trajectories of the ground-truth token, per-position loss curves, and a discrete loss measured at commit time. Second, the discrete loss allows assessing the correctness of recovery via cumulative discrete loss. We further analyse which tokens break the reconstructions and find a sharp categorical asymmetry: space-prefixed, high-frequency function words in dense regions of the embedding matrix dominate the failures, while content-bearing tokens are recovered almost perfectly. On 10-token C4 prompts the exact-match rate rises from 66.9% to 97.5% (mean similarity 0.994) as the candidate window is widened, confirming that most errors are recoverable near-misses rather than genuine ambiguities. A comparison with the released SIPIT reference situates these findings: per-step hard projection is faster, but the continuous formulation is what makes the optimisation observable and its failures detectable. The results show that last-layer hidden states of GPT-2 are as sensitive as the original text.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Hidden-state inversion on C4 (first 100 prompts)](<https://www.sota2.com/research/sota/hidden-state-inversion-on-c4-first-100-prompts.md>): Hidden-state inversion; C4 (first 100 prompts); EM: 100.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.00852>)

[PDF](<https://arxiv.org/pdf/2607.00852.pdf>)