AutoDW: Automatic Data Wrangling Leveraging Large Language Models

Data wrangling is a critical yet often labor-intensive process, essential for transforming raw data into formats suitable for downstream tasks such as machine learning or data analysis. Traditional data wrangling methods can be time-consuming, resource-intensive, and prone to errors, limiting the ef...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM International Conference on Automated Software Engineering : [proceedings] pp. 2041 - 2052
Main Authors: Liu, Lei, Hasegawa, So, Sampat, Shailaja Keyur, Xenochristou, Maria, Chen, Wei-Peng, Kato, Takashi, Kakibuchi, Taisei, Asai, Tatsuya
Format: Conference Proceeding
Language:English
Published: ACM 27.10.2024
Subjects:
Automation
Data models
Data Wrangling
Large language models
Limiting
Machine learning
Performance evaluation
Reproducibility of results
Software engineering
Source coding
Transforms
ISSN:2643-1572
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Data wrangling is a critical yet often labor-intensive process, essential for transforming raw data into formats suitable for downstream tasks such as machine learning or data analysis. Traditional data wrangling methods can be time-consuming, resource-intensive, and prone to errors, limiting the efficiency and effectiveness of subsequent downstream tasks. In this paper, we introduce AutoDW: an end-to-end solution for automatic data wrangling that leverages the power of Large Language Models (LLMs) to enhance automation and intelligence in data preparation. AutoDW distinguishes itself through several innovative features, including comprehensive automation that minimizes human intervention, the integration of LLMs to enable advanced data processing capabilities, and the generation of source code for the entire wrangling process, ensuring transparency and reproducibility. These advancements position AutoDW as a superior alternative to existing data wrangling tools, offering significant improvements in efficiency, accuracy, and flexibility. Through detailed performance evaluations, we demonstrate the effectiveness of AutoDW for data wrangling. We also discuss our experience and lessons learned from the industrial deployment of AutoDW, showcasing its potential to transform the landscape of automated data preparation.
ISSN:2643-1572
DOI:10.1145/3691620.3695267