# KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance

Source: [KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance](<https://www.sota2.com/research/paper/68fa7bef1e209131cea8b6da>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-11T05:00:04.810Z

Authors: Kuangshi Ai, Jonathan A. Karr Jr, Meng Jiang, Nitesh V. Chawla, Chaoli Wang

Published: 2025-10-07T02:29:13.000Z

## Abstract

We present Knowledge Extraction on OMIn (KEO), a domain-specific knowledge extraction and reasoning framework with large language models (LLMs) in safety-critical contexts. Using the Operations and Maintenance Intelligence (OMIn) dataset, we construct a QA benchmark spanning global sensemaking and actionable maintenance tasks. KEO builds a structured Knowledge Graph (KG) and integrates it into a retrieval-augmented generation (RAG) pipeline, enabling more coherent, dataset-wide reasoning than traditional text-chunk RAG. We evaluate locally deployable LLMs (Gemma-3, Phi-4, Mistral-Nemo) and employ stronger models (GPT-4o, Llama-3.3) as judges. Experiments show that KEO markedly improves global sensemaking by revealing patterns and system-level insights, while text-chunk RAG remains effective for fine-grained procedural tasks requiring localized retrieval. These findings underscore the promise of KG-augmented LLMs for secure, domain-specific QA and their potential in high-stakes reasoning. The code is available at https://github.com/JonathanKarr33/keo.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [83 global sensemaking questions](<https://www.sota2.com/research/sota/83-global-sensemaking-questions.md>): Global sensemaking; 83 global sensemaking questions; Overall Score: 4.87.

- [Knowledge-to-action Question Answering on 50 knowledge-to-action questions](<https://www.sota2.com/research/sota/knowledge-to-action-question-answering-on-50-knowledge-to-action-questions.md>): Knowledge-to-action question answering; 50 knowledge-to-action questions; Overall Score: 4.18.

- [Knowledge-to-Action QA on 50 Knowledge-to-Action Questions (gold-standard)](<https://www.sota2.com/research/sota/knowledge-to-action-qa-on-50-knowledge-to-action-questions-gold-standard.md>): Knowledge-to-action question answering; 50 knowledge-to-action questions gold-standard; ROUGE-1: 29.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.05524>)

[PDF](<https://arxiv.org/pdf/2510.05524.pdf>)