# On Understanding Attention-Based In-Context Learning for Categorical Data

Source: [On Understanding Attention-Based In-Context Learning for Categorical Data](<https://www.sota2.com/research/paper/68fa78621e209131cea7816c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:37:17.197Z

Authors: Aaron T. Wang, William Convertino, Xiang Cheng, Ricardo Henao, Lawrence Carin

Published: 2024-05-27T15:03:21.000Z

## Abstract

In-context learning based on attention models is examined for data with categorical outcomes, with inference in such models viewed from the perspective of functional gradient descent (GD). We develop a network composed of attention blocks, with each block employing a self-attention layer followed by a cross-attention layer, with associated skip connections. This model can exactly perform multi-step functional GD inference for in-context inference with categorical observations. We perform a theoretical analysis of this setup, generalizing many prior assumptions in this line of work, including the class of attention mechanisms for which it is appropriate. We demonstrate the framework empirically on synthetic data, image classification and language generation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.17248>)

[PDF](<https://arxiv.org/pdf/2405.17248.pdf>)