GAME-RL: Generating Adversarial Malware Examples Against API Call Based Detection via Reinforcement Learning

The adversarial example presents new security threats to trustworthy detection systems. In the context of evading dynamic detection based on API call sequences, a practical approach involves inserting perturbing API calls to modify these sequences. The type of inserted API calls and their insertion...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on dependable and secure computing Vol. 22; no. 5; pp. 5431 - 5447
Main Authors:	Zhan, Dazhi, Liu, Xin, Bai, Wei, Li, Wei, Guo, Shize, Pan, Zhisong
Format:	Journal Article
Language:	English
Published:	Washington IEEE 01.09.2025 IEEE Computer Society
Subjects:	adversarial example API call Application programming interface Application programming interfaces Closed box Computational modeling Cybersecurity dynamic analysis Feature extraction Games Generative adversarial networks Greedy algorithms IAT hooking Insertion Malware Operating systems Reinforcement learning reinforcement learning (RL) Training
ISSN:	1545-5971, 1941-0018
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	The adversarial example presents new security threats to trustworthy detection systems. In the context of evading dynamic detection based on API call sequences, a practical approach involves inserting perturbing API calls to modify these sequences. The type of inserted API calls and their insertion locations are crucial for generating an effective adversarial API call sequence. Existing methods either optimize the inserted API calls while neglecting the insertion positions or treat these optimizations as separate processes. This can lead to inefficient attacks that insert a large number of unnecessary API calls. To address this issue, we propose a novel reinforcement learning (RL) framework, dubbed GAME-RL, which simultaneously optimizes both the perturbing APIs and their insertion positions. Specifically, we define malware modification through IAT (Import Address Table) hooking as a sequential decision-making process. We introduce an invalid action masking and an auto-regressive policy head within the RL framework, ensuring the feasibility of IAT hooking and capturing the inherent relationship between factors. GAME-RL learns more effective evasion strategies, taking into account functionality preservation and the black-box setting. We conduct comprehensive experiments on various target models, demonstrating that GAME-RL significantly improves the evasion rate while maintaining acceptable levels of adversarial overhead.
AbstractList	The adversarial example presents new security threats to trustworthy detection systems. In the context of evading dynamic detection based on API call sequences, a practical approach involves inserting perturbing API calls to modify these sequences. The type of inserted API calls and their insertion locations are crucial for generating an effective adversarial API call sequence. Existing methods either optimize the inserted API calls while neglecting the insertion positions or treat these optimizations as separate processes. This can lead to inefficient attacks that insert a large number of unnecessary API calls. To address this issue, we propose a novel reinforcement learning (RL) framework, dubbed GAME-RL, which simultaneously optimizes both the perturbing APIs and their insertion positions. Specifically, we define malware modification through IAT (Import Address Table) hooking as a sequential decision-making process. We introduce an invalid action masking and an auto-regressive policy head within the RL framework, ensuring the feasibility of IAT hooking and capturing the inherent relationship between factors. GAME-RL learns more effective evasion strategies, taking into account functionality preservation and the black-box setting. We conduct comprehensive experiments on various target models, demonstrating that GAME-RL significantly improves the evasion rate while maintaining acceptable levels of adversarial overhead.
Author	Li, Wei Zhan, Dazhi Liu, Xin Guo, Shize Pan, Zhisong Bai, Wei
Author_xml	– sequence: 1 givenname: Dazhi orcidid: 0000-0003-2766-3405 surname: Zhan fullname: Zhan, Dazhi organization: Army Engineering University of PLA, Nanjing, China – sequence: 2 givenname: Xin surname: Liu fullname: Liu, Xin organization: Army Engineering University of PLA, Nanjing, China – sequence: 3 givenname: Wei surname: Bai fullname: Bai, Wei organization: Army Engineering University of PLA, Nanjing, China – sequence: 4 givenname: Wei surname: Li fullname: Li, Wei organization: Army Academy of Armored Forces, Beijing, China – sequence: 5 givenname: Shize surname: Guo fullname: Guo, Shize organization: Army Engineering University of PLA, Nanjing, China – sequence: 6 givenname: Zhisong orcidid: 0000-0001-8615-7313 surname: Pan fullname: Pan, Zhisong email: panzhisong@aeu.edu.cn organization: Army Engineering University of PLA, Nanjing, China
BookMark	eNp9kEtLAzEQx4Mo-PwAgoeA563JNtlNvK1trUJFqXpepsmsRLbZmsTXt3dLexAPMoeZw__B_A7Jru88EnLK2YBzpi-exo-jQc5yORjKoiiZ2iEHXAueMcbVbn9LITOpS75PDmN8ZSwXSosD0k6ru0k2n13SKXoMkJx_oZX9wBAhOGjpHbSfEJBOvmC5ajHS6gWcj4lWD7d0BG1LryCipWNMaJLrPP1wQOfofNMFg0v0ic4Qgu-Dj8leA23Ek-0-Is_Xk6fRTTa7n96Oqllmcl2krFxIUVptpAQhikZwaU3ecGuBMVFou7BKmX70AqUy3BotddNoa5nVvNFmeETON7mr0L29Y0z1a_cefF9ZD3NRCtUDkr2Kb1QmdDEGbOpVcEsI3zVn9RpqvYZar6HWW6i9p_zjMS7B-u0UwLX_Os82ToeIv5q0UiXnwx_ug4bF
CODEN	ITDSCM
CitedBy_id	crossref_primary_10_3390_act14090434 crossref_primary_10_3390_electronics14122422
Cites_doi	10.1007/978-3-030-86514-6_3 10.1145/3611643.3616356 10.1007/978-3-319-50127-7_11 10.1038/s42256-019-0138-9 10.1109/DSN.2019.00020 10.1145/3427228.3427230 10.1007/978-3-030-00470-5_23 10.1109/SPW.2018.00009 10.32473/flairs.v35i.130584 10.1609/aaai.v31i1.10804 10.1609/aaai.v34i01.5474 10.1145/3473039 10.1145/3326362 10.1016/j.asoc.2021.107506 10.1109/TDSC.2022.3153844 10.1007/s41635-017-0013-2 10.1109/TIFS.2023.3345640 10.1007/s10489-022-03523-2 10.1109/IJCNN52387.2021.9534432 10.1109/SP40000.2020.00073 10.1016/j.cose.2020.102118 10.1109/TC.2023.3339955 10.1145/3427228.3427242 10.1609/aaai.v32i1.11409 10.1109/TDSC.2023.3324265 10.1109/ICASSP.2013.6638293 10.1145/3411508.3421374 10.48550/arXiv.1812.05905 10.1109/BigData47090.2019.9006464 10.1109/DSN58367.2023.00027 10.1155/2015/659101 10.1109/WCNC.2013.6555301 10.1007/978-3-030-41579-2_2 10.18653/v1/n16-1174 10.1109/COMST.2022.3225137 10.1007/978-3-319-76941-7_71 10.1109/MSR52588.2021.00076 10.1109/TDSC.2018.2884928 10.1109/ICAIIC.2019.8669079 10.24963/ijcai.2019/382 10.1145/3052973.3053009 10.1109/SPW.2018.00020 10.1109/TIFS.2023.3333567 10.1007/978-981-19-8991-9_29 10.1109/SPW53761.2021.00021 10.1145/3133956.3133978 10.1109/EISIC.2017.21 10.1016/j.asoc.2021.107505
ContentType	Journal Article
Copyright	Copyright IEEE Computer Society 2025
Copyright_xml	– notice: Copyright IEEE Computer Society 2025
DBID	97E RIA RIE AAYXX CITATION JQ2
DOI	10.1109/TDSC.2025.3566708
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Xplore CrossRef ProQuest Computer Science Collection
DatabaseTitle	CrossRef ProQuest Computer Science Collection
DatabaseTitleList	ProQuest Computer Science Collection
Database_xml	– sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Computer Science
EISSN	1941-0018
EndPage	5447
ExternalDocumentID	10_1109_TDSC_2025_3566708 10988711
Genre	orig-research
GrantInformation_xml	– fundername: National Natural Science Foundation of China grantid: 62076251 funderid: 10.13039/501100001809
GroupedDBID	.4S .DC 0R~ 29I 4.4 5GY 5VS 6IK 7WY 8FE 8FG 8FL 8R4 8R5 97E AAJGR AARMG AASAJ AAWTH ABAZT ABJCF ABQJQ ABUWG ABVLG ACGFO ACIWK AENEX AETIX AFKRA AGQYO AGSQL AHBIQ AIBXA AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ARAPS ARCSS ATWAV AZQEC BEFXN BENPR BEZIV BFFAM BGLVJ BGNUA BKEBE BPEOZ BPHCQ CCPQU CS3 DU5 DWQXO EBS EDO EJD FRNLG GNUQQ HCIFZ HZ~ IEDLZ IFIPE IPLJI ITG ITH JAVBF K60 K6V K6~ K7- L6V LAI M0C M43 M7S O9- OCL P2P P62 PHGZM PHGZT PQBIZ PQBZA PQGLB PQQKQ PROAC PTHSS PUEGO Q2X RIA RIE RNI RNS RZB AAYXX AFFHD CITATION JQ2
ID	FETCH-LOGICAL-c296t-7b547d9c55a446f415dc2f1dda00469dbd88c8c89be58c1dc959ff9dd0d91f9c3
IEDL.DBID	RIE
ISICitedReferencesCount	2
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001564268700025&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1545-5971
IngestDate	Thu Nov 13 16:21:22 EST 2025 Tue Nov 18 22:24:21 EST 2025 Sat Nov 29 07:33:24 EST 2025 Wed Sep 10 07:40:43 EDT 2025
IsPeerReviewed	false
IsScholarly	true
Issue	5
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c296t-7b547d9c55a446f415dc2f1dda00469dbd88c8c89be58c1dc959ff9dd0d91f9c3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0001-8615-7313 0000-0003-2766-3405
PQID	3247486705
PQPubID	27603
PageCount	17
ParticipantIDs	crossref_primary_10_1109_TDSC_2025_3566708 proquest_journals_3247486705 crossref_citationtrail_10_1109_TDSC_2025_3566708 ieee_primary_10988711
PublicationCentury	2000
PublicationDate	2025-09-01
PublicationDateYYYYMMDD	2025-09-01
PublicationDate_xml	– month: 09 year: 2025 text: 2025-09-01 day: 01
PublicationDecade	2020
PublicationPlace	Washington
PublicationPlace_xml	– name: Washington
PublicationTitle	IEEE transactions on dependable and secure computing
PublicationTitleAbbrev	TDSC
PublicationYear	2025
Publisher	IEEE IEEE Computer Society
Publisher_xml	– name: IEEE – name: IEEE Computer Society
References	ref57 ref12 ref15 ref59 ref58 ref52 ref11 ref55 ref10 ref17 Dai (ref16) 2018 ref18 ref46 Li (ref63) 2023 ref45 ref48 ref42 ref41 ref44 ref43 ref49 ref8 ref7 ref4 ref3 Anderson (ref19) 2017; 2017 ref5 Raff (ref9) 2018 Catak (ref53) 2019 Anderson (ref6) 2018 ref35 ref34 ref37 ref36 ref31 ref30 Rosenberg (ref62) 2019 ref33 ref32 ref2 Goodfellow (ref13) 2015 ref1 ref39 ref38 Grosse (ref61) 2016 Metz (ref50) 2017; 3 Szegedy (ref14) 2014 ref24 ref23 ref26 ref25 Demontis (ref56) 2019 ref20 ref64 ref22 ref21 Schulman (ref51) 2017 ref28 ref27 ref29 Hu (ref40) 2018 Jordaney (ref54) 2017 Molnar (ref47) 2020 ref60
References_xml	– year: 2019 ident: ref53 article-title: A benchmark API call dataset for windows PE malware classification – start-page: 1181 volume-title: Proc. 32nd USENIX Secur. Symp. year: 2023 ident: ref63 article-title: Black-box adversarial example attack towards FCG based android malware detection under incomplete feature information – ident: ref22 doi: 10.1007/978-3-030-86514-6_3 – ident: ref64 doi: 10.1145/3611643.3616356 – ident: ref32 doi: 10.1007/978-3-319-50127-7_11 – ident: ref46 doi: 10.1038/s42256-019-0138-9 – ident: ref12 doi: 10.1109/DSN.2019.00020 – ident: ref28 doi: 10.1145/3427228.3427230 – ident: ref29 doi: 10.1007/978-3-030-00470-5_23 – year: 2017 ident: ref51 article-title: Proximal policy optimization algorithms – ident: ref15 doi: 10.1109/SPW.2018.00009 – ident: ref48 doi: 10.32473/flairs.v35i.130584 – year: 2019 ident: ref62 article-title: Defense methods against adversarial examples for recurrent neural networks – ident: ref44 doi: 10.1609/aaai.v31i1.10804 – ident: ref5 doi: 10.1609/aaai.v34i01.5474 – ident: ref33 doi: 10.1145/3473039 – ident: ref59 doi: 10.1145/3326362 – ident: ref41 doi: 10.1016/j.asoc.2021.107506 – ident: ref24 doi: 10.1109/TDSC.2022.3153844 – year: 2018 ident: ref6 article-title: EMBER: An open dataset for training static PE malware machine learning models – ident: ref26 doi: 10.1007/s41635-017-0013-2 – ident: ref43 doi: 10.1109/TIFS.2023.3345640 – ident: ref4 doi: 10.1007/s10489-022-03523-2 – ident: ref60 doi: 10.1109/IJCNN52387.2021.9534432 – ident: ref18 doi: 10.1109/SP40000.2020.00073 – ident: ref23 doi: 10.1016/j.cose.2020.102118 – ident: ref25 doi: 10.1109/TC.2023.3339955 – ident: ref3 doi: 10.1145/3427228.3427242 – ident: ref2 doi: 10.1609/aaai.v32i1.11409 – start-page: 321 volume-title: Proc. 28th USENIX Secur. Symp. (USENIX Secur. 19) year: 2019 ident: ref56 article-title: Why do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks – ident: ref42 doi: 10.1109/TDSC.2023.3324265 – volume: 2017 start-page: 1 year: 2017 ident: ref19 article-title: Evading machine learning malware detection publication-title: Black Hat – volume-title: Proc. Int. Conf. Learn. Represent year: 2015 ident: ref13 article-title: Explaining and harnessing adversarial examples – start-page: 245 volume-title: Proc. AAAI Conf. Artif. Intell. year: 2018 ident: ref40 article-title: Black-box attacks against RNN based malware detection algorithms – volume: 3 start-page: 1137 year: 2017 ident: ref50 article-title: Discrete sequential prediction of continuous actions for deep RL publication-title: J. Mach. Learn. Res. – ident: ref7 doi: 10.1109/ICASSP.2013.6638293 – ident: ref38 doi: 10.1145/3411508.3421374 – ident: ref52 doi: 10.48550/arXiv.1812.05905 – ident: ref11 doi: 10.1109/BigData47090.2019.9006464 – ident: ref30 doi: 10.1109/DSN58367.2023.00027 – ident: ref8 doi: 10.1155/2015/659101 – ident: ref57 doi: 10.1109/WCNC.2013.6555301 – ident: ref27 doi: 10.1007/978-3-030-41579-2_2 – ident: ref45 doi: 10.18653/v1/n16-1174 – start-page: 625 volume-title: Proc. 26th USENIX Secur. Symp. year: 2017 ident: ref54 article-title: Transcend: Detecting concept drift in malware classification models – ident: ref1 doi: 10.1109/COMST.2022.3225137 – ident: ref17 doi: 10.1007/978-3-319-76941-7_71 – ident: ref58 doi: 10.1109/MSR52588.2021.00076 – ident: ref10 doi: 10.1109/TDSC.2018.2884928 – volume-title: Proc. Int. Conf. Learn. Represent year: 2014 ident: ref14 article-title: Intriguing properties of neural networks – ident: ref36 doi: 10.1109/ICAIIC.2019.8669079 – ident: ref49 doi: 10.24963/ijcai.2019/382 – volume-title: Interpretable Machine Learning year: 2020 ident: ref47 – ident: ref39 doi: 10.1145/3052973.3053009 – ident: ref31 doi: 10.1109/SPW.2018.00020 – start-page: 268 volume-title: Proc. AAAI Conf. Artif. Intell. year: 2018 ident: ref9 article-title: Malware detection by eating a whole EXE – start-page: 1115 volume-title: Proc. 35th Int. Conf. Mach. Learn. year: 2018 ident: ref16 article-title: Adversarial attack on graph structured data – ident: ref34 doi: 10.1109/TIFS.2023.3333567 – ident: ref35 doi: 10.1007/978-981-19-8991-9_29 – ident: ref20 doi: 10.1109/SPW53761.2021.00021 – ident: ref55 doi: 10.1145/3133956.3133978 – ident: ref37 doi: 10.1109/EISIC.2017.21 – year: 2016 ident: ref61 article-title: Adversarial perturbations against deep neural networks for malware classification – ident: ref21 doi: 10.1016/j.asoc.2021.107505
SSID	ssj0024894
Score	2.4214067
Snippet	The adversarial example presents new security threats to trustworthy detection systems. In the context of evading dynamic detection based on API call...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	5431
SubjectTerms	adversarial example API call Application programming interface Application programming interfaces Closed box Computational modeling Cybersecurity dynamic analysis Feature extraction Games Generative adversarial networks Greedy algorithms IAT hooking Insertion Malware Operating systems Reinforcement learning reinforcement learning (RL) Training
Title	GAME-RL: Generating Adversarial Malware Examples Against API Call Based Detection via Reinforcement Learning
URI	https://ieeexplore.ieee.org/document/10988711 https://www.proquest.com/docview/3247486705
Volume	22
WOSCitedRecordID	wos001564268700025&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE/IET Electronic Library customDbUrl: eissn: 1941-0018 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0024894 issn: 1545-5971 databaseCode: RIE dateStart: 20040101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF60ePBifVSsVtmDJyE1fWw26632oYItpVboLewrpRBSadrqz3d2s9WCKEguOeyGkG8nM7M7830IXRMuJA009YgGawKP73usKQhkKUrWBDd8kNKKTdDBIJxM2NA1q9teGK21LT7TVXNrz_LVXK7MVhlYOAObMJ28u5TSvFnrm1gvtKqHJiTwIEquuSNMmHM77ry0IRWsk2oDohdqpCS3nJBVVfnxK7b-pVf855sdogMXSOJWjvwR2tHpMSpuRBqws9kTlDy0-l1v9HyHc4ZpU-aMrQxzxs3iw32evPOFxt0PbpiCM9ya8hkEjbg1fMJtniT4Hjydwh29tGVbKV7POB5pS7kq7e4idiyt0xJ67XXH7UfPSSx4ss6CpUcFaVLFJCEc8sIYvLmS9bimFLeJsxIqDCVcTGgSSsCPERbHTClfsVrMZOMUFdJ5qs8QDuM65conOmjoJmchh1CnIQLIh4QIVeyXkb_55pF0_ONGBiOJbB7is8jAFBmYIgdTGd18TXnLyTf-GlwyuGwNzCEpo8oG2cjZZxZBGEkN16BPzn-ZdoH2zdPzcrIKKiwXK32J9uR6OcsWV3bpfQLEA9Xk
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF6kCnqxPipWq-7Bk5CapNkm6632YcW2lFqht7CvlEJopS_9-c5utiqIguSSwy4J-XYyM7sz34fQNWFchFUVOkSBNYHHdx0acAJZihQeZ5oPUhixibDXi0Yj2rfN6qYXRillis9UWd-as3w5Eyu9VQYWTsEmdCfvNgkC38vatb6o9SKje6iDAgfiZM8eYsKs22HjuQ7JoE_KFYhfQi0m-c0NGV2VHz9j42Fa-X--2wHat6EkrmXYH6ItNT1C-Y1MA7ZWe4zSh1q36Qw6dzjjmNaFztgIMS-YXn64y9I3Nle4-c40V_AC18ZsAmEjrvUfcZ2lKb4HXydxQy1N4dYUrycMD5QhXRVmfxFbntZxAb20msN627EiC47waXXphJwEoaSCEAaZYQL-XAo_8aRkJnWWXEaRgItyRSIBCFJCk4RK6UrqJVRUTlBuOpuqU4SjxA-ZdImqVlTAaMQg2KnwKmREnEcycYvI3XzzWFgGci2EkcYmE3FprGGKNUyxhamIbj6nvGb0G38NLmhcvg3MICmi0gbZ2FroIoZAMtRsgy45-2XaFdptD7uduPPYezpHe_pJWXFZCeWW85W6QDtivZws5pdmGX4ADBjZKw
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GAME-RL%3A+Generating+Adversarial+Malware+Examples+Against+API+Call+Based+Detection+via+Reinforcement+Learning&rft.jtitle=IEEE+transactions+on+dependable+and+secure+computing&rft.au=Zhan%2C+Dazhi&rft.au=Liu%2C+Xin&rft.au=Bai%2C+Wei&rft.au=Li%2C+Wei&rft.date=2025-09-01&rft.issn=1545-5971&rft.eissn=1941-0018&rft.volume=22&rft.issue=5&rft.spage=5431&rft.epage=5447&rft_id=info:doi/10.1109%2FTDSC.2025.3566708&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TDSC_2025_3566708
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1545-5971&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1545-5971&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1545-5971&client=summon