# On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwarding

Source: [On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwarding](<https://www.sota2.com/research/paper/68fa7c471e209131cea8d5cd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-26T05:00:04.121Z

Authors: Jiahao Zhou, Chengliang Lin, Dingji Li, Mingkai Dong, Haibo Chen

Published: 2025-10-17T13:06:09.000Z

## Abstract

Semantic top-K selection with cross-encoder rerankers underpins on-device AI services, such as retrieval-augmented generation, agent memory, and personalized recommendation. However, its latency and memory demands dominate end-to-end budgets on edge hardware. Revisiting the objective of top-K selection, we reveal that only relative rankings matter, not exact per-candidate scores. We further observe sequence-level sparsity: relative rankings progressively stabilize in intermediate layers, enabling early pruning prior to completing full inference.   Building on this insight, we propose monolithic forwarding and develop a training-free inference system, PRISM. By maintaining a global view of all candidates, it reduces latency through progressive cluster pruning. It also bounds peak memory usage by strategically overlapping I/O with computation via overlapped layer streaming and chunked execution. We evaluate PRISM against state-of-the-art baselines on rerankers from 0.6 B to 8 B parameters across Apple M2 and RTX 5070. PRISM consistently reduces latency by up to 89.2% and peak memory by up to 91.3% in microbenchmarks, without compromising precision. Across three real-world on-device AI applications, PRISM lowers latency by 11.6%-51.0% and peak memory by 18.6%-77.8%, demonstrating substantial improvements in efficiency and deployability.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Information Retrieval on 18 datasets (BEIR, LoTTE, etc.)](<https://www.sota2.com/research/sota/information-retrieval-on-18-datasets-beir-lotte-etc.md>): Information Retrieval; 18 datasets (BEIR, LoTTE, etc.); P@1 Precision Loss Mean: -0.001.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.15620>)

[PDF](<https://arxiv.org/pdf/2510.15620.pdf>)
