# Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation

Source: [Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation](<https://www.sota2.com/research/paper/68fa76151e209131cea6b52e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T08:21:35.109Z

Authors: Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto

Published: 2024-02-12T22:35:40.000Z

## Abstract

Understanding textual description to generate code seems to be an achieved capability of instruction-following Large Language Models (LLMs) in zero-shot scenario. However, there is a severe possibility that this translation ability may be influenced by having seen target textual descriptions and the related code. This effect is known as Data Contamination.   In this study, we investigate the impact of Data Contamination on the performance of GPT-3.5 in the Text-to-SQL code-generating tasks. Hence, we introduce a novel method to detect Data Contamination in GPTs and examine GPT-3.5's Text-to-SQL performances using the known Spider Dataset and our new unfamiliar dataset Termite. Furthermore, we analyze GPT-3.5's efficacy on databases with modified information via an adversarial table disconnection (ATD) approach, complicating Text-to-SQL tasks by removing structural pieces of information from the database. Our results indicate a significant performance drop in GPT-3.5 on the unfamiliar Termite dataset, even with ATD modifications, highlighting the effect of Data Contamination on LLMs in Text-to-SQL translation tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-to-SQL on Spider (val)](<https://www.sota2.com/research/sota/text-to-sql-on-spider-val.md>): Text-to-SQL; Spider (val); Accuracy (Easy): not recorded.

- [Text-to-SQL on Termite (test)](<https://www.sota2.com/research/sota/text-to-sql-on-termite-test.md>): Text-to-SQL; Termite (test); Accuracy (Easy): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2402.08100>)

[PDF](<https://arxiv.org/pdf/2402.08100.pdf>)
