# LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications

Source: [LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications](<https://www.sota2.com/research/paper/69cb54fd0636e5bf4afc9e72>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-23T05:00:11.413Z

Authors: Alexandre Cristov\\~ao Maiorano

Published: 2026-03-28T18:03:32.000Z

## Abstract

We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow. The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract, then aggregates workflow success, policy compliance, groundedness, retrieval hit rate, cost, and p95 latency into scenario-weighted readiness scores with Pareto frontiers. We evaluate the harness on ticket-routing workflows and BEIR grounding tasks (SciFact and FiQA) with full Azure matrix coverage (162/162 valid cells across datasets, scenarios, retrieval depths, seeds, and models). Results show that readiness is not a single metric: on FiQA under sla-first at k=5, gpt-4.1-mini leads in readiness and faithfulness, while gpt-5.2 pays a substantial latency cost; on SciFact, models are closer in quality but still separable operationally. Ticket-routing regression gates consistently reject unsafe prompt variants, demonstrating that the harness can block risky releases instead of merely reporting offline scores. The result is a reproducible, operationally grounded framework for deciding whether an LLM or RAG system is ready to ship.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Information Retrieval on FiQA (Faithfulness and Hit@k)](<https://www.sota2.com/research/sota/information-retrieval-on-fiqa-faithfulness-and-hit-at-k.md>): Information Retrieval; FiQA; Faithfulness: not recorded.

- [Information Retrieval on SciFact (Faith., Hit@k, p95)](<https://www.sota2.com/research/sota/information-retrieval-on-scifact-faith-hit-at-k-p95.md>): Information Retrieval; SciFact; Faithfulness: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.27355>)

[PDF](<https://arxiv.org/pdf/2603.27355.pdf>)