# Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs

Source: [Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs](<https://www.sota2.com/research/paper/690d7cf12828b7cc6937cc8b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-06T05:01:13.325Z

Authors: Alberto Cattaneo, Carlo Luschi, Daniel Justus

Published: 2025-11-06T15:45:18.000Z

## Abstract

Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various solutions have been proposed, a comparison of methods is difficult due to the lack of challenging QA datasets with ground-truth targets for graph retrieval. We present SynthKGQA, an LLM-powered framework for generating high-quality Knowledge Graph Question Answering datasets from any Knowledge Graph, providing the full set of ground-truth facts in the KG to reason over questions. We show how, in addition to enabling more informative benchmarking of KG retrievers, the data produced with SynthKGQA also allows us to train better models.We apply SynthKGQA to Wikidata to generate GTSQA, a new dataset designed to test zero-shot generalization abilities of KG retrievers with respect to unseen graph structures and relation types, and benchmark popular solutions for KG-augmented LLMs on it.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2511.04473>)

[PDF](<https://arxiv.org/pdf/2511.04473.pdf>)
