LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems
Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tr...
Uloženo v:
| Vydáno v: | Proceedings / International Conference on Software Engineering s. 385 - 397 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
26.04.2025
|
| Témata: | |
| ISSN: | 1558-1225 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tracking). While various sUAS simulation tools exist to support developers, the entire process of creating, executing, and analyzing simulation tests remains a largely manual and cumbersome task. Developers must identify test scenarios, set up the simulation environment, integrate the System under Test (SuT) with simulation tools, formulate mission plans, and collect and analyze results. These labor-intensive tasks limit the ability of developers to conduct exhaustive testing across a wide range of scenarios. To alleviate this problem, in this paper, we propose Autosimtest, a Large Language Model (LLM)-driven framework, where multiple LLM agents collaborate to support the sUAS simulation testing process. This includes: (1) creating test scenarios that subject the SuT to unique environmental contexts; (2) preparing the simulation environment as per the test scenario; (3) generating diverse sUAS missions for the SuT to execute; and (4) analyzing simulation results and providing an interactive analytics interface. Further, the design of the framework is flexible for creating and testing scenarios for a variety of sUAS use cases, simulation tools, and SuT input requirements. We evaluated our approach by (a) conducting simulation testing of PX4 and ArduPilot flight-controller-based SuTs, (b) analyzing the performance of each agent, and (c) gathering feedback from sUAS developers. Our findings indicate that Autosimtest significantly improves the efficiency and scope of the sUAS testing process, allowing for more comprehensive and varied scenario evaluations while reducing the manual effort. |
|---|---|
| AbstractList | Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse weather conditions (such as wind, and fog), diverse settings (hilly terrain, or urban areas), and varying mission profiles (surveillance, tracking). While various sUAS simulation tools exist to support developers, the entire process of creating, executing, and analyzing simulation tests remains a largely manual and cumbersome task. Developers must identify test scenarios, set up the simulation environment, integrate the System under Test (SuT) with simulation tools, formulate mission plans, and collect and analyze results. These labor-intensive tasks limit the ability of developers to conduct exhaustive testing across a wide range of scenarios. To alleviate this problem, in this paper, we propose Autosimtest, a Large Language Model (LLM)-driven framework, where multiple LLM agents collaborate to support the sUAS simulation testing process. This includes: (1) creating test scenarios that subject the SuT to unique environmental contexts; (2) preparing the simulation environment as per the test scenario; (3) generating diverse sUAS missions for the SuT to execute; and (4) analyzing simulation results and providing an interactive analytics interface. Further, the design of the framework is flexible for creating and testing scenarios for a variety of sUAS use cases, simulation tools, and SuT input requirements. We evaluated our approach by (a) conducting simulation testing of PX4 and ArduPilot flight-controller-based SuTs, (b) analyzing the performance of each agent, and (c) gathering feedback from sUAS developers. Our findings indicate that Autosimtest significantly improves the efficiency and scope of the sUAS testing process, allowing for more comprehensive and varied scenario evaluations while reducing the manual effort. |
| Author | Vierhauser, Michael Agrawal, Ankit Zhang, Bohan Duvvuru, Venkata Sai Aswath |
| Author_xml | – sequence: 1 givenname: Venkata Sai Aswath surname: Duvvuru fullname: Duvvuru, Venkata Sai Aswath email: venkatasaiaswath.duvvuru@slu.edu organization: Saint Louis University,Department of Computer Science,Saint Louis,USA – sequence: 2 givenname: Bohan surname: Zhang fullname: Zhang, Bohan email: bohan.zhang.1@slu.edu organization: Saint Louis University,Department of Computer Science,Saint Louis,USA – sequence: 3 givenname: Michael surname: Vierhauser fullname: Vierhauser, Michael email: michael.vierhauser@uibk.ac.at organization: University of Innsbruck,Department of Computer Science,Innsbruck,Austria – sequence: 4 givenname: Ankit surname: Agrawal fullname: Agrawal, Ankit email: ankit.agrawal.1@slu.edu organization: Saint Louis University,Department of Computer Science,Saint Louis,USA |
| BookMark | eNotkM1OwkAYAFejiYC8AYd9geL-t9-xQUSSGg-FmwlZtl_JmnZruouGt5dET3OaOcyU3IUhICELzpacM3jaruq11lLlS8GEXjImhLwhc8ihkJJrpg3wWzLhWhcZF0I_kGmMn4wxowAm5KOq3rLyhCFF-jz6bwy0PKehtwkbWvv-3Nnkh0B3GJMPJ2pDQ8tgu0v0kQ4tjb3tOroPbsSfq1Hi6G1H60tM2MdHct_aLuL8nzOyf1nvVq9Z9b7Zrsoqs8KwlGGhjnkLTnEnZCFdbnKhmrzhiqNCAQ24VrW8YA60saClORp9dC1IzpQyQs7I4q_rEfHwNfrejpfDdY-AApj8BXWqVSk |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICSE55347.2025.00223 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798331505691 |
| EISSN | 1558-1225 |
| EndPage | 397 |
| ExternalDocumentID | 11029890 |
| Genre | orig-research |
| GroupedDBID | -~X .4S .DC 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-a260t-e84b7f9c41c2383c76724d7d141e4e29d9cf4f180c956a9536b65bcf931044623 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001538318100030&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 01:40:09 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a260t-e84b7f9c41c2383c76724d7d141e4e29d9cf4f180c956a9536b65bcf931044623 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_11029890 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-April-26 |
| PublicationDateYYYYMMDD | 2025-04-26 |
| PublicationDate_xml | – month: 04 year: 2025 text: 2025-April-26 day: 26 |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Conference on Software Engineering |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0006499 |
| Score | 2.2897186 |
| Snippet | Thorough simulation testing is crucial for validating the correct behavior of small Uncrewed Aerial Systems (sUAS) across multiple scenarios, including adverse... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 385 |
| SubjectTerms | AI for SE Autonomous aerial vehicles Manuals Planning Simulation Simulation Testing Software engineering sUAS Surveillance Testing Urban areas Usability Wind |
| Title | LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems |
| URI | https://ieeexplore.ieee.org/document/11029890 |
| WOSCitedRecordID | wos001538318100030&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3LSsNAFB20uHBVHxXfzMLt2M4jmWRZaotCLYW20oVQJvOQQk2kTfX3vTNNqxsX7kIgBO48zrlz55yL0J1minMRW-Iy4YjIpCaJdxwCbFOSiZi1gpnOS18OBsl0mg4rsXrQwlhrw-Uze-8fQy3fFHrtj8qaAFXeMBwy9H0p441Ya7ftxsDdK20cbaXNp86oG0VcSMgBWTg38R2JfnVQCQDSq__z10eo8SPFw8MdyByjPZufoPq2FwOuluYpeu33n0nb66RW-GHptzDcXpcF8FFr8Gj-XnXpwmPvqpG_YZUbvDUkwYXDq3e1WOBJDiTyC75oh4mJKz_zBpr0uuPOI6k6JxAF-UlJbAJRd6kWVAMkcy1jiLuRhgpqhWWpSbUTjiYtDemR8hXcLI4y7VIge5AfMn6GanmR23OEuXZUUeupTio2VbZIU81ZlhjKrLlADR-t2cfGHGO2DdTlH--v0KEfEF-QYfE1qpXLtb1BB_qznK-Wt2FIvwF23aE0 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3LTgIxFG2MmugKHxjfduG2Qjud15IgBOJASADDwoTM9GFIYMbAoL_vbSnoxoW7ySSTSW4f99zennMQehQs9TweKKIzrgnPQkEiozgEuS0NGQ9Y3YrpvCZhvx9NJvHAkdUtF0YpZS-fqSfzaHv5shBrc1RWg1RlBMOhQj8w1lmOrrXbeANA744dR-txrdsctnzf4yFUgcyenBhPol8eKjaFtCv__PkJqv6Q8fBgl2ZO0Z7Kz1Bl68aA3eI8R29J0iMNw5Ra4eel2cRwY10WgEiVxMPZwvl04ZHR1cjfcZpLvJUkwYXGq0U6n-NxDjDyC75o2KmJnaJ5FY3brVGzQ5x3AkmhQimJiiDuOhacCkjKnggDiLwMJeVUccViGQvNNY3qAgqk1PRws8DPhI4B7kGFyLwLtJ8XubpE2BOaplQZsBPzTZ_NF1R4LIskZUpeoaqJ1vRjI48x3Qbq-o_3D-ioM-ol06Tbf7lBx2ZwTHuGBbdov1yu1R06FJ_lbLW8t8P7DQxypH0 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=LLM-Agents+Driven+Automated+Simulation+Testing+and+Analysis+of+small+Uncrewed+Aerial+Systems&rft.au=Duvvuru%2C+Venkata+Sai+Aswath&rft.au=Zhang%2C+Bohan&rft.au=Vierhauser%2C+Michael&rft.au=Agrawal%2C+Ankit&rft.date=2025-04-26&rft.pub=IEEE&rft.eissn=1558-1225&rft.spage=385&rft.epage=397&rft_id=info:doi/10.1109%2FICSE55347.2025.00223&rft.externalDocID=11029890 |