# Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation

Source: [Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation](<https://www.sota2.com/research/paper/68fa71291e209131cea4f94d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:59:40.801Z

Authors: Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky

Published: 2023-11-01T17:43:35.000Z

## Abstract

An ideal length-extrapolatable Transformer language model can handle sequences longer than the training length without any fine-tuning. Such long-context utilization capability relies heavily on a flexible positional embedding design. Upon investigating the flexibility of existing large pre-trained Transformer language models, we find that the T5 family deserves a closer look, as its positional embeddings capture rich and flexible attention patterns. However, T5 suffers from the dispersed attention issue: the longer the input sequence, the flatter the attention distribution. To alleviate the issue, we propose two attention alignment strategies via temperature scaling. Our findings show improvement on the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. This suggests that a flexible positional embedding design and attention alignment can go a long way toward Transformer length extrapolation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2311.00684>)

[PDF](<https://arxiv.org/pdf/2311.00684.pdf>)
