Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study.
Saved in:
| Title: | Exploring ChatGPT's Potential in Java API Method Recommendation: An Empirical Study. |
|---|---|
| Authors: | Wang, Ye, Xue, Weihao, Huang, Qiao, Jiang, Bo, Zhang, Hua |
| Source: | Journal of Software: Evolution & Process; Jan2025, Vol. 37 Issue 1, p1-20, 20p |
| Subject Terms: | LANGUAGE models, CHATGPT, COMPUTER software development, EMPIRICAL research, APPLICATION program interfaces |
| Abstract: | As software development grows increasingly complex, application programming interface (API) plays a significant role in enhancing development efficiency and code quality. However, the explosive growth in the number of APIs makes it impossible for developers to become familiar with all of them. In actual development scenarios, developers may spend a significant amount of time searching for suitable APIs, which could severely impact the development process. Recently, the OpenAI's large language model (LLM) based application—ChatGPT has shown exceptional performance across various software development tasks, responding swiftly to instructions and generating high‐quality textual responses, suggesting its potential in API recommendation tasks. Thus, this paper presents an empirical study to investigate the performance of ChatGPT in query‐based API recommendation tasks. Specifically, we utilized the existing benchmark APIBENCH‐Q and the newly constructed dataset as evaluation datasets, selecting the state‐of‐the‐art models BIKER and MULAREC for comparison with ChatGPT. Our research findings demonstrate that ChatGPT outperforms existing approaches in terms of success rate, mean reciprocal rank (MRR), and mean average precision (MAP). Through a manual examination of samples in which ChatGPT exceeds baseline performance and those where it provides incorrect answers, we further substantiate ChatGPT's advantages over the baselines and identify several issues contributing to its suboptimal performance. To address these issues and enhance ChatGPT's recommendation capabilities, we employed two strategies: (1) utilizing a more advanced LLM (GPT‐4) and (2) exploring a new approach—MACAR, which is based on the Chain of Thought methodology. The results indicate that both strategies are effective. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Software: Evolution & Process is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Complementary Index |
Full Text Finder 
Nájsť tento článok vo Web of Science Be the first to leave a comment!