# Study of positional encoding approaches for Audio Spectrogram Transformers

Source: [Study of positional encoding approaches for Audio Spectrogram Transformers](<https://www.sota2.com/research/paper/68fa70d01e209131cea4d61c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:50:52.513Z

Authors: Leonardo Pepino, Pablo Riera, Luciana Ferrer

Published: 2021-10-13T19:20:20.000Z

## Abstract

Transformers have revolutionized the world of deep learning, specially in the field of natural language processing. Recently, the Audio Spectrogram Transformer (AST) was proposed for audio classification, leading to state of the art results in several datasets. However, in order for ASTs to outperform CNNs, pretraining with ImageNet is needed. In this paper, we study one component of the AST, the positional encoding, and propose several variants to improve the performance of ASTs trained from scratch, without ImageNet pretraining. Our best model, which incorporates conditional positional encodings, significantly improves performance on Audioset and ESC-50 compared to the original AST.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2110.06999>)

[PDF](<https://arxiv.org/pdf/2110.06999.pdf>)