SQUAB: Evaluating LLM robustness to Ambiguous and Unanswerable Questions in Semantic Parsing
Saved in:
| Title: | SQUAB: Evaluating LLM robustness to Ambiguous and Unanswerable Questions in Semantic Parsing |
|---|---|
| Authors: | Papicchio, Simone, Cagliero, Luca, Papotti, Paolo |
| Contributors: | Papicchio, Simone, Cagliero, Luca, Papotti, Paolo |
| Publisher Information: | Association for Computational Linguistics |
| Publication Year: | 2025 |
| Collection: | PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino) |
| Subject Terms: | Text2SQL, Ambiguity, Dynamic Benchmark, Industry, Unanswerable, Text-to-SQL |
| Description: | Large Language Models (LLMs) have demonstrated robust performance in Semantic Parsing (SP) for well-defined queries with unambiguous intent and answerable responses. However, practical user questions frequently deviate from these ideal conditions, challenging the applicability of existing benchmarks. To address this issue, we introduce SQUAB, an automatic dataset generator of Ambiguous and Unanswerable questions. SQUAB generates complex, annotated SP tests using a blend of SQL and LLM capabilities. Results show that SQUAB reduces test generation costs by up to 99% compared to human-based solutions while aligning with real-world question patterns. Furthermore, these tests challenge LLM performance while revealing disparities between public and proprietary datasets. This highlights the need for a dynamic, automatic dataset generator as SQUAB. The code is designed for user extension to accommodate new ambiguous and unanswerable patterns and is available at https://github.com/spapicchio/squab. |
| Document Type: | conference object |
| File Description: | ELETTRONICO |
| Language: | English |
| Relation: | ispartofbook:Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing; Conference on Empirical Methods in Natural Language Processing; firstpage:17937; lastpage:17957; numberofpages:21; https://hdl.handle.net/11583/3005408 |
| DOI: | 10.18653/v1/2025.emnlp-main.906 |
| Availability: | https://hdl.handle.net/11583/3005408 https://doi.org/10.18653/v1/2025.emnlp-main.906 https://aclanthology.org/2025.emnlp-main.906/ |
| Rights: | info:eu-repo/semantics/openAccess ; license:Creative commons ; license uri:http://creativecommons.org/licenses/by/4.0/ |
| Accession Number: | edsbas.B3CB5026 |
| Database: | BASE |
Nájsť tento článok vo Web of Science Be the first to leave a comment!