# Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding

Source: [Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding](<https://www.sota2.com/research/paper/68fa79d81e209131cea7fb27>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-11-05T05:02:10.101Z

Authors: Ebrahim Feghhi, Shreyas Kaasyap, Nima Hadidi, Jonathan C. Kao

Published: 2025-07-03T17:02:54.000Z

## Abstract

Speech neuroprostheses aim to restore communication for people with severe paralysis by decoding speech directly from neural activity. To accelerate algorithmic progress, a recent benchmark released intracranial recordings from a paralyzed participant attempting to speak, along with a baseline decoding algorithm. Prior work on the benchmark showed impressive accuracy gains. However, these gains increased computational costs and were not demonstrated in a real-time decoding setting. Here, we make three contributions that pave the way towards accurate, efficient, and real-time neural speech decoding. First, we incorporate large amounts of time-masking during training. On average, over $50\\%$ of each trial is masked. Second, we replace the gated recurrent unit (GRU) architecture used in the baseline algorithm with a compact Transformer. The Transformer architecture uses $83\\%$ fewer parameters, cuts peak GPU memory usage by $52\\%$, and is significantly faster to calibrate relative to the GRU. Third, we design a lightweight variant of an existing test-time adaptation method developed for decoding handwriting from neural activity. Our variant adapts the model using multiple time-masked augmentations of a single trial and requires only one gradient step per trial. Together, these contributions reduce word error rate by over $20\\%$ and effectively mitigate performance degradations across held-out days in a real-time decoding setting while substantially lowering computational costs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speech Decoding on Brain-to-Text 2024 (hold-out)](<https://www.sota2.com/research/sota/speech-decoding-on-brain-to-text-2024-hold-out.md>): Speech Decoding; Brain-to-Text 2024 (hold-out); WER: 5.68.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.02800>)

[PDF](<https://arxiv.org/pdf/2507.02800.pdf>)