# A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI

Source: [A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI](<https://www.sota2.com/research/paper/69b8e0090636e5bf4afbe9ef>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-18T05:00:04.466Z

Authors: Karim Helwani, Hoang Do, James Luan, Sriram Srinivasan

Published: 2026-03-10T21:17:17.000Z

## Abstract

We present a real-time front-end for voice-based conversational AI to enable natural turn-taking in two-speaker scenarios by combining primary speaker segmentation with hierarchical End-of-Turn (EOT) detection. To operate robustly in multi-speaker environments, the system continuously identifies and tracks the primary user, ensuring that downstream EOT decisions are not confounded by background conversations. The tracked activity segments are fed to a hierarchical, causal EOT model that predicts the immediate conversational state by independently analyzing per-speaker speech features from both the primary speaker and the bot. Simultaneously, the model anticipates near-future states ($t{+}10/20/30$\\,ms) through probabilistic predictions that are aware of the conversation partner's speech. Task-specific knowledge distillation compresses wav2vec~2.0 representations (768\\,D) into a compact MFCC-based student (32\\,D) for efficient deployment. The system achieves 82\\% multi-class frame-level F1 and 70.6\\% F1 on Backchannel detection, with 69.3\\% F1 on a binary Final vs.\\ Others task. On an end-to-end turn-detection benchmark, our model reaches 87.7\\% recall vs.\\ 58.9\\% for Smart Turn~v3 while keeping a median detection latency of 36\\,ms versus 800--1300\\,ms. Despite using only 1.14\\,M parameters, the proposed model matches or exceeds transformer-based baselines while substantially reducing latency and memory footprint, making it suitable for edge deployment.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [End-of-Turn Detection on 357 conversation samples](<https://www.sota2.com/research/sota/end-of-turn-detection-on-357-conversation-samples.md>): End-of-Turn Detection; 357 conversation samples; Latency (ms): 36.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.13379>)

[PDF](<https://arxiv.org/pdf/2603.13379.pdf>)