Automated Testing for Service-Oriented Architecture: Leveraging Large Language Models for Enhanced Service Composition

This article explores the application of Large Language Models (LLMs), including proprietary models such as OpenAI's ChatGPT 4o and ChatGPT 4o-mini, Anthropic's Claude 3.5 Sonnet and Claude 3.7 Sonnet, and Google's Gemini 1.5 Pro, Gemini 2.0 Flash, and Gemini 2.0 Flash-Lite, as well a...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access Vol. 13; pp. 89627 - 89640
Main Authors: Altin, Mahsun, Mutlu, Behcet, Kilinc, Deniz, Cakir, Altan
Format: Journal Article
Language:English
Published: Piscataway IEEE 2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Accuracy
AI-driven automation
Application programming interface
Artificial intelligence
Automation
Chatbots
Completion time
Complexity theory
Computer architecture
Flight
Indexing
Large language models
Parameters
Real time
Real-time systems
Scalability
scalability in API testing
Service oriented architecture
Software development management
Software reliability
Software systems
Software testing
software validation
ISSN:2169-3536, 2169-3536
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract This article explores the application of Large Language Models (LLMs), including proprietary models such as OpenAI's ChatGPT 4o and ChatGPT 4o-mini, Anthropic's Claude 3.5 Sonnet and Claude 3.7 Sonnet, and Google's Gemini 1.5 Pro, Gemini 2.0 Flash, and Gemini 2.0 Flash-Lite, as well as open-source alternatives including Qwen2.5-14B-Instruct-1M, and commercially accessed models such as DeepSeek R1 and DeepSeek V3, which were tested via APIs despite having open-source variants, to automate validation and verification in Application Programming Interface (API) testing within a Service-Oriented Architecture (SOA). Our system compares internal responses from the Enuygun Web Server against third-party API outputs in both JSON and XML formats, validating critical parameters such as flight prices, baggage allowances, and seat availability. We generated 100 diverse test scenarios across varying complexities (1-4 flight results) by randomly altering request and response parameters. Experimental results show that Google Gemini 2.0 Flash achieved high accuracy (up to 99.98%) with the lowest completion time (85.34 seconds), while Qwen2.5-14B-Instruct-1M exhibited limited capability in processing complex formats. Models such as OpenAI's ChatGPT and Anthropic's Claude Sonnet models also demonstrated strong performance in single-flight validation scenarios, making them suitable for low-latency, high-precision tasks. Our findings indicate that some open-source models can offer promising cost-effective alternatives, though performance significantly varies. This integration of LLMs reduced manual workload, improved test scalability, and enabled real-time validation across large-scale datasets. As LLM technologies mature, we anticipate further advances in automation, accuracy, and efficiency in software validation systems.
AbstractList This article explores the application of Large Language Models (LLMs), including proprietary models such as OpenAI’s ChatGPT 4o and ChatGPT 4o-mini, Anthropic’s Claude 3.5 Sonnet and Claude 3.7 Sonnet, and Google’s Gemini 1.5 Pro, Gemini 2.0 Flash, and Gemini 2.0 Flash-Lite, as well as open-source alternatives including Qwen2.5-14B-Instruct-1M, and commercially accessed models such as DeepSeek R1 and DeepSeek V3, which were tested via APIs despite having open-source variants, to automate validation and verification in Application Programming Interface (API) testing within a Service-Oriented Architecture (SOA). Our system compares internal responses from the Enuygun Web Server against third-party API outputs in both JSON and XML formats, validating critical parameters such as flight prices, baggage allowances, and seat availability. We generated 100 diverse test scenarios across varying complexities (1-4 flight results) by randomly altering request and response parameters. Experimental results show that Google Gemini 2.0 Flash achieved high accuracy (up to 99.98%) with the lowest completion time (85.34 seconds), while Qwen2.5-14B-Instruct-1M exhibited limited capability in processing complex formats. Models such as OpenAI’s ChatGPT and Anthropic’s Claude Sonnet models also demonstrated strong performance in single-flight validation scenarios, making them suitable for low-latency, high-precision tasks. Our findings indicate that some open-source models can offer promising cost-effective alternatives, though performance significantly varies. This integration of LLMs reduced manual workload, improved test scalability, and enabled real-time validation across large-scale datasets. As LLM technologies mature, we anticipate further advances in automation, accuracy, and efficiency in software validation systems.
Author Kilinc, Deniz
Altin, Mahsun
Mutlu, Behcet
Cakir, Altan
Author_xml – sequence: 1
  givenname: Mahsun
  orcidid: 0000-0002-5285-2593
  surname: Altin
  fullname: Altin, Mahsun
  email: altinma21@itu.edu.tr
  organization: Department of Computer Engineering, Istanbul Technical University, İstanbul, Türkiye
– sequence: 2
  givenname: Behcet
  orcidid: 0000-0003-1054-8558
  surname: Mutlu
  fullname: Mutlu, Behcet
  organization: Wingie Enuygun Group, İstanbul, Türkiye
– sequence: 3
  givenname: Deniz
  orcidid: 0000-0002-2336-8831
  surname: Kilinc
  fullname: Kilinc, Deniz
  organization: Department of Computer Engineering, University of Bakirçay, İzmir, Türkiye
– sequence: 4
  givenname: Altan
  surname: Cakir
  fullname: Cakir, Altan
  organization: Department of Data Science and Analytics, Istanbul Technical University, İstanbul, Türkiye
BookMark eNp9UU2P0zAQtdAisSz7C-AQiXOKP-I45lZFBVYq2kOXszXrjLOu2rg4TqX99zibIhAHfBg_jd97mvF7S66GMCAh7xldMUb1p3Xbbna7FadcroRUTOvqFbnmrNalkKK--gu_IbfjuKf5NLkl1TU5r6cUjpCwKx5wTH7oCxdiscN49hbL--hxmB_X0T75hDZNET8XWzxjhH5mbyH2mOvQT5DB99DhYXzx2AxPMNisvZgVbTiewuiTD8M78trBYcTby31DfnzZPLTfyu3917t2vS1txVgqK6q5rhrMW9VcNAxErWjHqs4y5-rGCcdBVso92toCANM2o445Ck5Z5FLckLvFtwuwN6fojxCfTQBvXhoh9gZi8vaAxlaPnYK6oY6ySlQSJHdCNNJ1tnFWqOz1cfE6xfBzyp9l9mGKQx7fCM54I5nSNLP0wrIxjGNEZ6xPMO-cIviDYdTMqZklNTOnZi6pZa34R_t74v-rPiwqj4h_FIxSJbkWvwAQyqaJ
CODEN IAECCG
CitedBy_id crossref_primary_10_3390_buildings15132273
Cites_doi 10.3390/app14052074
10.1109/TSE.2023.3334955
10.1145/3597503.3639150
10.1145/3641289
10.1109/OJCOMS.2024.3456549
10.1108/00251740710718935
10.1016/j.future.2024.05.034
10.3390/pr10091782
10.3390/app12094369
10.1109/ICNSurv.2013.6548579
10.1109/SANER60148.2024.00041
10.3390/electronics13173563
10.1145/3643733
10.1007/s12599-009-0074-z
10.1109/ACCESS.2021.3073041
10.1109/TKDE.2022.3148237
10.1145/3659677.3659749
10.1145/3611643.3613083
10.1109/QRS60937.2023.00029
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
DBID 97E
ESBDL
RIA
RIE
AAYXX
CITATION
7SC
7SP
7SR
8BQ
8FD
JG9
JQ2
L7M
L~C
L~D
DOA
DOI 10.1109/ACCESS.2025.3571994
DatabaseName IEEE Xplore (IEEE)
IEEE Xplore Open Access Journals
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Engineered Materials Abstracts
METADEX
Technology Research Database
Materials Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DOAJ: Directory of Open Access Journal (DOAJ)
DatabaseTitle CrossRef
Materials Research Database
Engineered Materials Abstracts
Technology Research Database
Computer and Information Systems Abstracts – Academic
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Advanced Technologies Database with Aerospace
METADEX
Computer and Information Systems Abstracts Professional
DatabaseTitleList Materials Research Database
Database_xml – sequence: 1
  dbid: DOA
  name: DOAJ Directory of Open Access Journals
  url: https://www.doaj.org/
  sourceTypes: Open Website
– sequence: 2
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2169-3536
EndPage 89640
ExternalDocumentID oai_doaj_org_article_c4bd7a680f014345a52f3385fdc8fc37
10_1109_ACCESS_2025_3571994
11007529
Genre orig-research
GrantInformation_xml – fundername: Wingie Enuygun Group
– fundername: ITU-BAP Doctoral Projects 2025
GroupedDBID 0R~
4.4
5VS
6IK
97E
AAJGR
ABAZT
ABVLG
ACGFS
ADBBV
AGSQL
ALMA_UNASSIGNED_HOLDINGS
BCNDV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
EJD
ESBDL
GROUPED_DOAJ
IPLJI
JAVBF
KQ8
M43
M~E
O9-
OCL
OK1
RIA
RIE
RNS
AAYXX
CITATION
7SC
7SP
7SR
8BQ
8FD
JG9
JQ2
L7M
L~C
L~D
ID FETCH-LOGICAL-c411t-4092948e57162381a3670d14dc1ff68f3f2a547fbc6caaa19cbc6d1f0af7ce253
IEDL.DBID DOA
ISICitedReferencesCount 1
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001498048200024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 2169-3536
IngestDate Fri Oct 03 12:51:36 EDT 2025
Sat Nov 01 15:15:59 EDT 2025
Tue Nov 18 22:32:16 EST 2025
Sat Nov 29 07:54:26 EST 2025
Wed Aug 27 01:51:27 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Language English
License https://creativecommons.org/licenses/by/4.0/legalcode
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c411t-4092948e57162381a3670d14dc1ff68f3f2a547fbc6caaa19cbc6d1f0af7ce253
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0003-1054-8558
0000-0002-5285-2593
0000-0002-2336-8831
OpenAccessLink https://doaj.org/article/c4bd7a680f014345a52f3385fdc8fc37
PQID 3212851790
PQPubID 4845423
PageCount 14
ParticipantIDs doaj_primary_oai_doaj_org_article_c4bd7a680f014345a52f3385fdc8fc37
crossref_primary_10_1109_ACCESS_2025_3571994
ieee_primary_11007529
crossref_citationtrail_10_1109_ACCESS_2025_3571994
proquest_journals_3212851790
PublicationCentury 2000
PublicationDate 20250000
2025-00-00
20250101
2025-01-01
PublicationDateYYYYMMDD 2025-01-01
PublicationDate_xml – year: 2025
  text: 20250000
PublicationDecade 2020
PublicationPlace Piscataway
PublicationPlace_xml – name: Piscataway
PublicationTitle IEEE access
PublicationTitleAbbrev Access
PublicationYear 2025
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref12
Liu (ref29) 2024
ref15
ref14
ref31
ref11
ref10
(ref25) 2024
ref2
ref17
ref16
Li (ref18) 2024
Yang (ref30) 2025
Andriyanto (ref22) 2021
ref23
ref21
Hurst (ref24) 2024
Roche (ref1) 2014
Wang (ref19) 2024
Guo (ref28) 2025
ref8
ref7
ref9
ref4
(ref26) 2025
ref3
Jin (ref20) 2024
ref6
ref5
Georgiev (ref27) 2024
References_xml – ident: ref6
  doi: 10.3390/app14052074
– ident: ref10
  doi: 10.1109/TSE.2023.3334955
– year: 2021
  ident: ref22
  article-title: X-driven methodologies for SOA system development—A survey
  publication-title: arXiv:2109.01805
– ident: ref13
  doi: 10.1145/3597503.3639150
– year: 2024
  ident: ref20
  article-title: From LLMs to LLM-based agents for software engineering: A survey of current, challenges and future
  publication-title: arXiv:2408.02479
– ident: ref7
  doi: 10.1145/3641289
– year: 2014
  ident: ref1
  article-title: Integrating service orientated architecture design principles into software as a service applications
– ident: ref21
  doi: 10.1109/OJCOMS.2024.3456549
– volume-title: Claude 3.7 Sonnet System Card
  year: 2025
  ident: ref26
– ident: ref4
  doi: 10.1108/00251740710718935
– ident: ref17
  doi: 10.1016/j.future.2024.05.034
– ident: ref2
  doi: 10.3390/pr10091782
– ident: ref5
  doi: 10.3390/app12094369
– year: 2024
  ident: ref27
  article-title: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  publication-title: arXiv:2403.05530
– ident: ref3
  doi: 10.1109/ICNSurv.2013.6548579
– year: 2024
  ident: ref19
  article-title: A roadmap for software testing in open collaborative development environments
  publication-title: arXiv:2406.05438
– ident: ref16
  doi: 10.1109/SANER60148.2024.00041
– ident: ref31
  doi: 10.3390/electronics13173563
– volume-title: arXiv:2501.15383
  year: 2025
  ident: ref30
  article-title: Qwen2.5–1M technical report
– ident: ref12
  doi: 10.1145/3643733
– year: 2024
  ident: ref18
  article-title: Model-enhanced LLM-driven VUI testing of VPA apps
  publication-title: arXiv:2407.02791
– ident: ref23
  doi: 10.1007/s12599-009-0074-z
– ident: ref14
  doi: 10.1109/ACCESS.2021.3073041
– ident: ref15
  doi: 10.1109/TKDE.2022.3148237
– ident: ref8
  doi: 10.1145/3659677.3659749
– ident: ref11
  doi: 10.1145/3611643.3613083
– year: 2024
  ident: ref24
  article-title: GPT-4o system card
  publication-title: arXiv:2410.21276
– volume-title: arXiv:2412.19437
  year: 2024
  ident: ref29
  article-title: DeepSeek-V3 technical report
– volume-title: Claude 3.5 Sonnet Model Card Addendum
  year: 2024
  ident: ref25
– year: 2025
  ident: ref28
  article-title: DeepSeek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning
  publication-title: arXiv:2501.12948
– ident: ref9
  doi: 10.1109/QRS60937.2023.00029
SSID ssj0000816957
Score 2.3412054
Snippet This article explores the application of Large Language Models (LLMs), including proprietary models such as OpenAI's ChatGPT 4o and ChatGPT 4o-mini,...
This article explores the application of Large Language Models (LLMs), including proprietary models such as OpenAI’s ChatGPT 4o and ChatGPT 4o-mini,...
SourceID doaj
proquest
crossref
ieee
SourceType Open Website
Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 89627
SubjectTerms Accuracy
AI-driven automation
Application programming interface
Artificial intelligence
Automation
Chatbots
Completion time
Complexity theory
Computer architecture
Flight
Indexing
Large language models
Parameters
Real time
Real-time systems
Scalability
scalability in API testing
Service oriented architecture
Software development management
Software reliability
Software systems
Software testing
software validation
SummonAdditionalLinks – databaseName: IEEE Electronic Library (IEL)
  dbid: RIE
  link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Lb9QwEB7RigMceBaxpSAfOJI2fsV2b8uqFYdV4VBQb5Z3YgNSlUXd3f5-PI53KUIgcYmsxHaczNjz8PgbgLeiky2GNjRcBtuoiLEJKLBJQodF4NK2BUrpy9xcXNirK_epHlYvZ2FijCX4LB5Tsezl90vckKvshODNjBZuD_aM6cbDWjuHCmWQcNpUZCHeupPpbJY_ItuAQh9LbQgF9zfpU0D6a1aVP5biIl_OH__nyJ7Ao6pIsulI-adwLw7P4OEdeMHncDvdrJdZI409uyQwjeEryyoqq8tD85Egjunh9M5mwimbx8zdJXcRm1OYeL6OLk1GedOuV6WPs-FbCR3YdsZoXanxXwfw-fzscvahqXkWGlScr7MJmXUkZaMmNKkswQOBuvVc9chT6mySSQStTFpghyEE7jCXep7akAxGoeUL2B-WQ3wJTGm0BnW2-TpUjmtrZecyswq-kCFxNwGx_f8eKwg55cK49sUYaZ0fieaJaL4SbQLvdo1-jBgc_67-ngi7q0oA2uVGppiv89GjWvQmdLZNBHCodNAiZWtdpx5tQmkmcEBU_vW-SuAJHG35xNfZvvIyy39bsM4O_9LsFTygIY6-myPYX99s4mu4j7fr76ubN4WRfwLb3PGb
  priority: 102
  providerName: IEEE
Title Automated Testing for Service-Oriented Architecture: Leveraging Large Language Models for Enhanced Service Composition
URI https://ieeexplore.ieee.org/document/11007529
https://www.proquest.com/docview/3212851790
https://doaj.org/article/c4bd7a680f014345a52f3385fdc8fc37
Volume 13
WOSCitedRecordID wos001498048200024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVAON
  databaseName: DOAJ Directory of Open Access Journals
  customDbUrl:
  eissn: 2169-3536
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000816957
  issn: 2169-3536
  databaseCode: DOA
  dateStart: 20130101
  isFulltext: true
  titleUrlDefault: https://www.doaj.org/
  providerName: Directory of Open Access Journals
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2169-3536
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000816957
  issn: 2169-3536
  databaseCode: M~E
  dateStart: 20130101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LT9wwELYQ6qE9VC2l6gJFPvTYlPiV2L1tV4s4bCkHQNws78QulVBA7MKxv70zjhcWVSqXXqwoceyMZ-yZccbfMPZJNqqGUIdKqGArHSFWASRUSZowD0LZOkMpnc_a42N7ceFO1lJ9UUzYAA88DNwB6HnXhsbWiZDotAlGJnSrTOrAJlD5HHndujVnKq_BVjTOtAVmSNTuYDyZIEXoEErzRZmWIHGfqKKM2F9SrPy1Lmdlc_iGvS5WIh8PX_eWbcR-i71aww58x-7Hd8trNDdjx08JKaP_ydH-5GXuVz8Iv5gejtf-FHzls4iimxMT8RnFgGM57FdySop2tchtTPvLHBewaozTolGCu7bZ2eH0dHJUlSQKFWghlugfogGkbTQEFYXqORBiWyd0ByKlxiaVZDC6TXNoIIQgHOBVJ1IdUgtRGvWebfbXffzAuDZgWzDo0DWgnTDWqsahJEoxVyEJN2JyNZ4eCsI4Jbq48tnTqJ0fmOCJCb4wYcQ-P7x0MwBs_Lv6N2LUQ1VCx843UGZ8kRn_nMyM2Dax-bE_ChUxEgnYW_Hdl6m88AqVu81AZjv_o-9d9pLoGXZx9tjm8vYufmQv4H75a3G7n6UYy--_p_v5LOIf0yX2VQ
linkProvider Directory of Open Access Journals
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Lb9QwELagIAEHnq1YKOADR9LGr9jmtqxaFREWDgvqzfJObKhUZVF3t78fj-NdilCReomsxHacfGN7PB5_Q8hb3ogafO0rJrypZIBQeeBQRa783DNh6kyl9L3V06k5PbVfy2H1fBYmhJCdz8IBJvNefreANZrKDpHeTCtub5M7SkpeD8e1tiYVjCFhlS7cQqy2h-PJJH1GWgVydSCURh7cv-afTNNf4qr8MxjnGeb40Q3b9pg8LKokHQ_YPyG3Qv-UPLhCMPiMXI7Xq0XSSUNHZ0in0f-gSUmlZYCoviDJMT4cX9lOeE_bkOQ7Ry-iLTqKp-tg1KQYOe18mes46n9m54FNZRRHluIBtku-HR_NJidVibRQgWRslRaRSUuSJijkk0pzuEdat47JDliMjYkicq-kjnNowHvPLKRUx2Lto4bAldgjO_2iD88JlQqMhgSPaUBapowRjU3iytlc-MjsiPDN_3dQaMgxGsa5y8uR2roBNIeguQLaiLzbFvo1sHD8P_sHBHabFSm0842EmCs90oGcd9o3po5IcSiVVzym9bqKHZgIQo_ILqL8530F4BHZ38iJK_196UTSAExmO3txTbE35N7J7HPr2o_TTy_JfWzuYMnZJzuri3V4Re7C5epsefE6C_Vvjgj04g
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automated+Testing+for+Service-Oriented+Architecture%3A+Leveraging+Large+Language+Models+for+Enhanced+Service+Composition&rft.jtitle=IEEE+access&rft.au=Altin%2C+Mahsun&rft.au=Mutlu%2C+Behcet&rft.au=Kilinc%2C+Deniz&rft.au=Cakir%2C+Altan&rft.date=2025&rft.issn=2169-3536&rft.eissn=2169-3536&rft.volume=13&rft.spage=89627&rft.epage=89640&rft_id=info:doi/10.1109%2FACCESS.2025.3571994&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_ACCESS_2025_3571994
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2169-3536&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2169-3536&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2169-3536&client=summon