On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability
Large language models (LLMs) are increasingly used to generate software artifacts, such as source code, tests, and trace links. Requirements play a central role in shaping the input prompts that guide LLMs, as they are often used as part of the prompts to synthesize the artifacts. However, the impac...
Gespeichert in:
| Veröffentlicht in: | IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Technologies Results (Online) S. 51 - 55 |
|---|---|
| Hauptverfasser: | , , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
27.04.2025
|
| Schlagworte: | |
| ISSN: | 2832-7632 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Large language models (LLMs) are increasingly used to generate software artifacts, such as source code, tests, and trace links. Requirements play a central role in shaping the input prompts that guide LLMs, as they are often used as part of the prompts to synthesize the artifacts. However, the impact of requirements formulation on LLM performance remains unclear. In this paper, we investigate the role of requirements smells- indicators of potential issues like ambiguity and inconsistency- when used in prompts for LLMs. We conducted experiments using two LLMs focusing on automated trace link generation between requirements and code. Our results show mixed outcomes: while requirements smells had a small but significant effect when predicting whether a requirement was implemented in a piece of code (i.e., a trace link exists), no significant effect was observed when tracing the requirements with the associated lines of code. These findings suggest that requirements smells can affect LLM performance in certain SE tasks but may not uniformly impact all tasks. We highlight the need for further research to understand these nuances and propose future work toward developing guidelines for mitigating the negative effects of requirements smells in AI-driven SE processes. |
|---|---|
| AbstractList | Large language models (LLMs) are increasingly used to generate software artifacts, such as source code, tests, and trace links. Requirements play a central role in shaping the input prompts that guide LLMs, as they are often used as part of the prompts to synthesize the artifacts. However, the impact of requirements formulation on LLM performance remains unclear. In this paper, we investigate the role of requirements smells- indicators of potential issues like ambiguity and inconsistency- when used in prompts for LLMs. We conducted experiments using two LLMs focusing on automated trace link generation between requirements and code. Our results show mixed outcomes: while requirements smells had a small but significant effect when predicting whether a requirement was implemented in a piece of code (i.e., a trace link exists), no significant effect was observed when tracing the requirements with the associated lines of code. These findings suggest that requirements smells can affect LLM performance in certain SE tasks but may not uniformly impact all tasks. We highlight the need for further research to understand these nuances and propose future work toward developing guidelines for mitigating the negative effects of requirements smells in AI-driven SE processes. |
| Author | Broccia, Giovanna Korn, Alexander Vogelsang, Andreas Ferrari, Alessio Fischbach, Jannik Arora, Chetan |
| Author_xml | – sequence: 1 givenname: Andreas surname: Vogelsang fullname: Vogelsang, Andreas email: vogelsang@cs.uni-koeln.de organization: University of Cologne,Cologne,Germany – sequence: 2 givenname: Alexander surname: Korn fullname: Korn, Alexander email: korn@cs.uni-koeln.de organization: University of Cologne,Cologne,Germany – sequence: 3 givenname: Giovanna surname: Broccia fullname: Broccia, Giovanna email: giovanna.broccia@isti.cnr.it organization: CNR-ISTI,Pisa,Italy – sequence: 4 givenname: Alessio surname: Ferrari fullname: Ferrari, Alessio email: alessio.ferrari@ucd.ie organization: University College Dublin,Dublin,Ireland – sequence: 5 givenname: Jannik surname: Fischbach fullname: Fischbach, Jannik email: jannik.fischbach@netlight.com organization: Netlight Consulting GmbH and Fortiss GmbH,Munich,Germany – sequence: 6 givenname: Chetan surname: Arora fullname: Arora, Chetan email: chetan.arora@monash.edu organization: Monash University,Melbourne,Australia |
| BookMark | eNot0M1OwzAQBGCDQKKUvgEH8wApXju2E25V1EKkiqK2nKvFWYug_JTYPfTtCYLTSKNPc5hbdtX1HTH2AGIOIPLHstgtk9dyuTVGaTmXQuq5EALMBZvlNs-UAq0sAFyyicyUTKxR8obNQvgamZIAFrIJ2206Hj-Jl-0RXeS951v6PtUDtdTFwHctNU3gdcffhr49xvDE96MuMNCvXZxi32Kkiu8HdIQfdVPH8x279tgEmv3nlL2vlvviJVlvnstisU5QWhETEs6CEuC9BpdiJZzXSMJIqvLUp0je5-hTMKh95sxYKl2lVlltyFVUqSm7_9utiehwHOoWh_NhfEeqPAX1A2hbVSE |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/ICSE-NIER66352.2025.00016 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9798331537111 |
| EISSN | 2832-7632 |
| EndPage | 55 |
| ExternalDocumentID | 11023941 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK OCL RIE RIL |
| ID | FETCH-LOGICAL-a270t-e0c71301ff51c4ad0cf5ae062ed94f4aeff9af416a5f8c6d9435d473756ecded3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001552151900011&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Jun 18 06:01:24 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a270t-e0c71301ff51c4ad0cf5ae062ed94f4aeff9af416a5f8c6d9435d473756ecded3 |
| PageCount | 5 |
| ParticipantIDs | ieee_primary_11023941 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-April-27 |
| PublicationDateYYYYMMDD | 2025-04-27 |
| PublicationDate_xml | – month: 04 year: 2025 text: 2025-April-27 day: 27 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Technologies Results (Online) |
| PublicationTitleAbbrev | ICSE-NIER |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003211718 |
| Score | 2.2966938 |
| Snippet | Large language models (LLMs) are increasingly used to generate software artifacts, such as source code, tests, and trace links. Requirements play a central... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 51 |
| SubjectTerms | Codes Focusing Guidelines Large language models LLMs Requirements Eng Smells Software Software engineering Source coding Traceability |
| Title | On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability |
| URI | https://ieeexplore.ieee.org/document/11023941 |
| WOSCitedRecordID | wos001552151900011&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA46RDypONH5gwhe49I2TVpvMjbcZY5NYbcRk_dgsLVj7QT_e5NsTi8evJXQQHlN-733ku_7CLnnklsvMMkUomHCAjD3FSkmokxjIjODwgSzCTUYZJNJPtyS1QMXBgDC4TN48JdhL9-WZu1bZe0oCk7ertjZV0puyFq7hkriShn3oz0kd1sdzXa_M-6yQb878qjqSVex759wb23-y0klAEnv-J-PcEKaP5Q8OtyBzSnZg-KMjF8K6hI42g9UR1oiHYE_2RtafhUdL2A-r-is8DMXy7p6pG5V0I4DLn_v07ouXb4KljrAMrAR7P5skrde97XzzLYuCUzHitcMuHGFJo8Q08gIbbnBVAOXMdhcoNCAmGt0eZdOMTPSDSapFSpRqQRjwSbnpFGUBVwQKhKMrPTyOSoV5l3kmKVulhauhtNKJZek6SMyXW6EMKbfwWj9MX5FjnzQ_eZLrK5Jo16t4YYcmI96Vq1uw-v7Ap6Pm9c |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwGA2iop5UnPjbCF7j0jZpWm8yNlacdWwTdhsx-QKDrRtrJ_jfm2RzevHgrYQGQtLmfd-XvPcQuqcx1U5gkghjFGEagNi_SBAWJNJEcaIMU95sQuR5Mhym3TVZ3XNhAMBfPoMH9-jP8vVMLV2prB4E3snbJjs7zjqLr-ham5JKZJMZu9Xuobu1kmY9a_SbJM-aPYerjnYVugoKdebmv7xUPJS0Dv85iCNU-yHl4e4Gbo7RFhQnqP9aYBvC4cyTHfHM4B64u72-6Ffi_hQmkxKPC9dzOq_KR2y_C9yw0OXefVpWMxuxgsYWshSsJLs_a-it1Rw02mTtk0BkKGhFgCqbatLAGB4oJjVVhkugcQg6ZYZJMCaVxkZekptExbYx4pqJSPAYlAYdnaLtYlbAGcIsMoGOnYCO4Ey9s9Qk3PaSzGZxUojoHNXcjIzmKymM0fdkXPzRfov224OXzqiT5c-X6MAtgDuKCcUV2q4WS7hGu-qjGpeLG7-UX_IRnyI |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Software+Engineering%3A+New+Ideas+and+Emerging+Technologies+Results+%28Online%29&rft.atitle=On+the+Impact+of+Requirements+Smells+in+Prompts%3A+The+Case+of+Automated+Traceability&rft.au=Vogelsang%2C+Andreas&rft.au=Korn%2C+Alexander&rft.au=Broccia%2C+Giovanna&rft.au=Ferrari%2C+Alessio&rft.date=2025-04-27&rft.pub=IEEE&rft.eissn=2832-7632&rft.spage=51&rft.epage=55&rft_id=info:doi/10.1109%2FICSE-NIER66352.2025.00016&rft.externalDocID=11023941 |