View in EDS

ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models.

Saved in:
Bibliographic Details
Title: ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models.
Authors: Ascoli, Benjamin G., Kandikonda, Yasoda Sai Ram, Choi, Jinho D.
Source: Future Internet; Aug2025, Vol. 17 Issue 8, p325, 16p
Subject Terms: LANGUAGE models, NATURAL language processing, OPEN source software, QUERY languages (Computer science), STATISTICAL accuracy
Abstract: The task of Text-to-SQL enables anyone to retrieve information from SQL databases using natural language. While this task has made substantial progress, the two primary evaluation metrics—Execution Accuracy (EXE) and Exact Set Matching Accuracy (ESM)—suffer from inherent limitations that can misrepresent performance. Specifically, ESM's rigid matching overlooks semantically correct but stylistically different queries, whereas EXE can overestimate correctness by ignoring structural errors that yield correct outputs. These shortcomings become especially problematic when assessing outputs from large language model (LLM)-based approaches without fine-tuning, which vary more in style and structure compared with their fine-tuned counterparts. Thus, we introduce a new metric, Enhanced Tree Matching (ETM), which mitigates these issues by comparing queries using both syntactic and semantic elements. Through evaluating nine LLM-based models, we show that EXE and ESM can produce false positive and negative rates as high as 23.0% and 28.9%, while ETM reduces these rates to 0.3% and 2.7%, respectively. We release our ETM script as open-source, offering the community a more robust and reliable approach to evaluating Text-to-SQL. [ABSTRACT FROM AUTHOR]
Copyright of Future Internet is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Complementary Index
Description
Abstract:The task of Text-to-SQL enables anyone to retrieve information from SQL databases using natural language. While this task has made substantial progress, the two primary evaluation metrics—Execution Accuracy (EXE) and Exact Set Matching Accuracy (ESM)—suffer from inherent limitations that can misrepresent performance. Specifically, ESM's rigid matching overlooks semantically correct but stylistically different queries, whereas EXE can overestimate correctness by ignoring structural errors that yield correct outputs. These shortcomings become especially problematic when assessing outputs from large language model (LLM)-based approaches without fine-tuning, which vary more in style and structure compared with their fine-tuned counterparts. Thus, we introduce a new metric, Enhanced Tree Matching (ETM), which mitigates these issues by comparing queries using both syntactic and semantic elements. Through evaluating nine LLM-based models, we show that EXE and ESM can produce false positive and negative rates as high as 23.0% and 28.9%, while ETM reduces these rates to 0.3% and 2.7%, respectively. We release our ETM script as open-source, offering the community a more robust and reliable approach to evaluating Text-to-SQL. [ABSTRACT FROM AUTHOR]
ISSN:19995903
DOI:10.3390/fi17080325