# From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis

Source: [From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis](<https://www.sota2.com/research/paper/69cb55020636e5bf4afc9ec7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-01T05:00:04.436Z

Authors: Eason Chen, Isabel Wang, Nina Yuan, Sophia Judicke, Kayla Beigh, Xinyi Tang

Published: 2026-03-28T23:04:36.000Z

## Abstract

Behavioral analysis of tutoring dialogues is essential for understanding student learning, yet manual coding remains a bottleneck. We present a methodology where LLM coding agents autonomously improve the prompts used by LLM classifiers to label educational dialogues. In each iteration, a coding agent runs the classifier against human-labeled validation data, analyzes disagreements, and proposes theory-grounded prompt modifications for researcher review. Applying this approach to 659 AI tutoring sessions across four experiments with three agents and three classifiers, 4-fold cross-validation on held-out data confirmed genuine improvement: the best agent achieved test $\\kappa=0.78$ (SD$=0.08$), matching human inter-rater reliability ($\\kappa=0.78$), at a cost of approximately \\$5--8 per agent. While development-set performance reached $\\kappa=0.91$--$0.93$, the cross-validated results represent our primary generalization claim. The iterative process also surfaced an undocumented labeling pattern: human coders consistently treated expressions of confusion as engagement rather than disengagement. Continued iteration beyond the optimum led to regression, underscoring the need for held-out validation. We release all prompts, iteration logs, and data.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Conversation Labeling on Cross-agent 80 labeled sessions (val)](<https://www.sota2.com/research/sota/conversation-labeling-on-cross-agent-80-labeled-sessions-val.md>): Conversation labeling; Cross-agent 80 labeled sessions (val); Overall Kappa: 0.93.

- [Labeling Task Performance on 80-session Set (4-fold CV)](<https://www.sota2.com/research/sota/labeling-task-performance-on-80-session-set-4-fold-cv.md>): Labeling task performance; 80-session set 4-fold cross-validation (test); Fold 0 Performance: 85.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.27440>)

[PDF](<https://arxiv.org/pdf/2603.27440.pdf>)
