Towards Autonomous Testing Agents via Conversational Large Language Models

Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, an...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM International Conference on Automated Software Engineering : [proceedings] pp. 1688 - 1693
Main Authors: Feldt, Robert, Kang, Sungmin, Yoon, Juyeon, Yoo, Shin
Format: Conference Proceeding
Language:English
Published: IEEE 11.09.2023
Subjects:
artificial intelligence
Automation
Drives
large language model
machine learning
Middleware
Oral communication
Software testing
Taxonomy
test automation
Testing
ISSN:2643-1572
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized "hallucination" of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.
AbstractList Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized "hallucination" of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.
Author Feldt, Robert
Yoon, Juyeon
Kang, Sungmin
Yoo, Shin
Author_xml – sequence: 1
  givenname: Robert
  surname: Feldt
  fullname: Feldt, Robert
  email: robert.feldt@chalmers.se
  organization: Chalmers University of Technology
– sequence: 2
  givenname: Sungmin
  surname: Kang
  fullname: Kang, Sungmin
  email: sungmin.kang@kaist.ac.kr
  organization: KAIST
– sequence: 3
  givenname: Juyeon
  surname: Yoon
  fullname: Yoon, Juyeon
  email: juyeon.yoon@kaist.ac.kr
  organization: KAIST
– sequence: 4
  givenname: Shin
  surname: Yoo
  fullname: Yoo, Shin
  email: shin.yoo@kaist.ac.kr
  organization: KAIST
BookMark eNotj1FLwzAUhaMouM39An3oH-i8SZp0eSxjTqXig_V53DY3pdIl0rQT_70FhcM533k5cJbsygdPjN1x2HAO5qF43ysthNkIEHIDwLPtBVub3GylAimM0dklWwidyZSrXNywZYyfAGou-YK9VOEbBxuTYhqDD6cwxaSiOHa-TYqW_BiTc4fJLvgzDRHHLnjskxKHlmb37YQzvAZLfbxl1w77SOv_XLGPx321e0rLt8PzrihTlBrG1BFZWYNooEFbK3TKOjJNpmuOwupG5k3jnHV8Vo1mS0ZJgFwYTiRQoVyx-7_djoiOX0N3wuHnyEHMjzXIX6w8UYo
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ASE56229.2023.00148
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library (IEL) (UW System Shared)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL) (UW System Shared)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798350329964
EISSN 2643-1572
EndPage 1693
ExternalDocumentID 10298360
Genre orig-research
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IM
6IN
6J9
AAJGR
AAWTH
ABLEC
ACREN
ADYOE
ADZIZ
AFYQB
ALMA_UNASSIGNED_HOLDINGS
AMTXH
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
M43
OCL
RIE
RIL
ID FETCH-LOGICAL-a360t-feed3b02c0cadb5af5dfe9c46b1a2d6c37ccffdf1df1ba98e953007291ee2a5a3
IEDL.DBID RIE
ISICitedReferencesCount 15
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001103357200135&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:32:41 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a360t-feed3b02c0cadb5af5dfe9c46b1a2d6c37ccffdf1df1ba98e953007291ee2a5a3
PageCount 6
ParticipantIDs ieee_primary_10298360
PublicationCentury 2000
PublicationDate 2023-Sept.-11
PublicationDateYYYYMMDD 2023-09-11
PublicationDate_xml – month: 09
  year: 2023
  text: 2023-Sept.-11
  day: 11
PublicationDecade 2020
PublicationTitle IEEE/ACM International Conference on Automated Software Engineering : [proceedings]
PublicationTitleAbbrev ASE
PublicationYear 2023
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0051577
ssib057256115
Score 2.3876717
Snippet Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test...
SourceID ieee
SourceType Publisher
StartPage 1688
SubjectTerms artificial intelligence
Automation
Drives
large language model
machine learning
Middleware
Oral communication
Software testing
Taxonomy
test automation
Testing
Title Towards Autonomous Testing Agents via Conversational Large Language Models
URI https://ieeexplore.ieee.org/document/10298360
WOSCitedRecordID wos001103357200135&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LawIxEA5VeujJPiytfZBDr9ua7CYxRxGlFBGhtniTbB5FKGtxV39_Z-Jqe-mhsITdPQ2ThMlMvu8bQh40ZMk5HB0SbV2GCQpL4NMmnknvpPDcRDGd97GaTHrzuZ7WZPXIhfHeR_CZf8TXeJfvVnaDpTLY4Vwj6aBBGkrJHVlrv3iEguDN2OHsC3FaqVpmiHX1U_91CKGeIzeFo6gpw44_vxqqxHgyav3TklPS_mHm0ekh5pyRI1-ck9a-NQOtd-oFeZlFOGxJ-5sKaQuQ39MZCmoUH7SPbKqSbpeGDhBzvi7rgiAdIywcxl0Jk2KftM-yTd5Gw9ngOanbJiQGLKqSACaA_7ntWuNyYYJwwWubyZwZ7qRNlbUhuMDgyY3ueS3SKCDOPEyNMOklaRarwl8RmjIrTda1kFWZzASep6LnlXUmZyEXMrsmbfTN4munjLHYu6Xzx_8bcoLuR7wFY7ekWa03_o4c2221LNf3cT6_ASecofY
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB60Cnqqj4pv9-A12k2ySfdYSqVqLAWj9Fb2KQVJpUn7-93ZptWLByEsSU7Dzi6zMzvf9wHccpclS3d0CLjSMSYoNHCfKjA0MTphJhSeTOc9S4fDznjMRzVY3WNhjDG--czc4au_y9cztcBSmdvhIUfQwTbsoHQWW8G11suHpS58U7o5_bpInaY10RBt8_vua98F-xDRKSHSmlLU_PklqeIjykPzn7YcQOsHm0dGm6hzCFumOILmWpyB1Hv1GJ5y3xBbku6iQuCCy_BJjpQaxQfpIp6qJMupID3sOp-XdUmQZNgY7sZVEZOgUtpn2YK3h37eGwS1cEIgnEVVYJ0JzgOhaiuhJROWaWu4ihNJRagTFaVKWastdY8UvGM4izyFODXOOUxEJ9AoZoU5BRJRlYi4rVxeJWJhQxmxjkmVFpJayZL4DFo4N5OvFTfGZD0t53_8v4G9Qf6STbLH4fMF7KMrsPuC0ktoVPOFuYJdtaym5fza-_YbyT-lQQ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=Towards+Autonomous+Testing+Agents+via+Conversational+Large+Language+Models&rft.au=Feldt%2C+Robert&rft.au=Kang%2C+Sungmin&rft.au=Yoon%2C+Juyeon&rft.au=Yoo%2C+Shin&rft.date=2023-09-11&rft.pub=IEEE&rft.eissn=2643-1572&rft.spage=1688&rft.epage=1693&rft_id=info:doi/10.1109%2FASE56229.2023.00148&rft.externalDocID=10298360