# Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts

Source: [Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts](<https://www.sota2.com/research/paper/69d33df57d0172140ec73651>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-07T05:00:04.608Z

Authors: Jinsook Lee, Kirk Vanacore, Zhuqian Zhou, Bakhtawar Ahtisham, Rene F. Kizilcec

Published: 2026-04-03T15:49:43.000Z

## Abstract

Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's $\\kappa$ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines ($\\kappa = 0.275$-$0.413$ and $0.160$-$0.410$). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7\\% to 62.0\\% on TalkMoves and 52.9\\% to 73.1\\% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Dialogue Annotation on TalkMoves (test)](<https://www.sota2.com/research/sota/dialogue-annotation-on-talkmoves-test.md>): Dialogue Annotation; TalkMoves (test); Cohen's Kappa: 58.

- [Dialogue Annotation on Eedi (test)](<https://www.sota2.com/research/sota/dialogue-annotation-on-eedi-test.md>): Dialogue Annotation; Eedi (test); Cohen's Kappa: 0.743.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.03127>)

[PDF](<https://arxiv.org/pdf/2604.03127.pdf>)