# Beyond One-shot: AI Agents for Learning in Field Experiments

Source: [Beyond One-shot: AI Agents for Learning in Field Experiments](<https://www.sota2.com/research/paper/6a1e6450dbd7150506f859ba>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:04:15.602Z

Authors: Junjie Luo, Ritu Agarwal, Gordon Gao

Published: 2026-06-01T16:30:42.000Z

## Abstract

Organizations routinely run experiments for A/B testing, yet the data generated from one experiment is underutilized to inform subsequent intervention design. Significant barriers exist to extracting actionable knowledge from prior experimental data to inform new interventions. We study whether tool-augmented agentic AI can automatically learn from experimental data to generate new interventions in subsequent experiments. Through two-stage field experiments in healthcare prescription messaging (693,139 patient visits), we compare a Human + Chatbot method (Stage 1: behavioral experts with conversational AI co-designing 13 message variants, 444,691 patient visits) against a Tool-Augmented Agentic AI method (Stage 2: AI autonomously extracting principles from Stage 1 data to generate 17 new variants, 248,448 patient visits). The Agentic AI method, equipped with analytical tools, structured Data-Information-Knowledge-Wisdom (DIKW) reasoning agents, and transparent evidence chains, produces superior interventions: the best AI-generated message achieved a 69.8% CTR (+6.5 percentage points over baseline). Critically, our results suggest that the value comes from domain-specific experimental data, not from general reasoning ability: frontier LLMs operating without experimental data failed to predict which interventions would succeed. The field experiments also revealed that general-purpose behavioral theories used for intervention design do not extend uniformly to specific healthcare contexts, motivating an agentic AI approach to theory audits at field-experiment scale. Our research shows that tool-augmented AI can learn from experimental data and generate improved domain-relevant interventions, transforming behavioral experimentation from one-shot evaluation into a scalable system for cumulative design learning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Click-Through Rate Prediction on Agentic AI Message Dataset (test)](<https://www.sota2.com/research/sota/click-through-rate-prediction-on-agentic-ai-message-dataset-test.md>): Click-Through Rate Prediction; Agentic AI Message Dataset (test); Coefficient: 0.0654.

- [Authentication Rate Prediction on Agentic AI Message (test)](<https://www.sota2.com/research/sota/authentication-rate-prediction-on-agentic-ai-message-test.md>): Authentication Rate Prediction; Agentic AI Message (test); Authentication Rate: 59.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.02458>)

[PDF](<https://arxiv.org/pdf/2606.02458.pdf>)