Guidance Design for Escape Flight Vehicle Using Evolution Strategy Enhanced Deep Reinforcement Learning
Guidance commands of flight vehicles are a series of data sets with fixed time intervals, thus guidance design constitutes a sequential decision problem and satisfies the basic conditions for using deep reinforcement learning (DRL). In this paper, we consider the scenario where the escape flight veh...
Uloženo v:
| Vydáno v: | arXiv.org |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Paper |
| Jazyk: | angličtina |
| Vydáno: |
Ithaca
Cornell University Library, arXiv.org
04.05.2024
|
| Témata: | |
| ISSN: | 2331-8422 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Guidance commands of flight vehicles are a series of data sets with fixed time intervals, thus guidance design constitutes a sequential decision problem and satisfies the basic conditions for using deep reinforcement learning (DRL). In this paper, we consider the scenario where the escape flight vehicle (EFV) generates guidance commands based on DRL and the pursuit flight vehicle (PFV) generates guidance commands based on the proportional navigation method. For the EFV, the objective of the guidance design entails progressively maximizing the residual velocity, subject to the constraint imposed by the given evasion distance. Thus an irregular dynamic max-min problem of extremely large-scale is formulated, where the time instant when the optimal solution can be attained is uncertain and the optimum solution depends on all the intermediate guidance commands generated before. For solving this problem, a two-step strategy is conceived. In the first step, we use the proximal policy optimization (PPO) algorithm to generate the guidance commands of the EFV. The results obtained by PPO in the global search space are coarse, despite the fact that the reward function, the neural network parameters and the learning rate are designed elaborately. Therefore, in the second step, we propose to invoke the evolution strategy (ES) based algorithm, which uses the result of PPO as the initial value, to further improve the quality of the solution by searching in the local space. Simulation results demonstrate that the proposed guidance design method based on the PPO algorithm is capable of achieving a residual velocity of 67.24 m/s, higher than the residual velocities achieved by the benchmark soft actor-critic and deep deterministic policy gradient algorithms. Furthermore, the proposed ES-enhanced PPO algorithm outperforms the PPO algorithm by 2.7\%, achieving a residual velocity of 69.04 m/s. |
|---|---|
| AbstractList | Guidance commands of flight vehicles are a series of data sets with fixed time intervals, thus guidance design constitutes a sequential decision problem and satisfies the basic conditions for using deep reinforcement learning (DRL). In this paper, we consider the scenario where the escape flight vehicle (EFV) generates guidance commands based on DRL and the pursuit flight vehicle (PFV) generates guidance commands based on the proportional navigation method. For the EFV, the objective of the guidance design entails progressively maximizing the residual velocity, subject to the constraint imposed by the given evasion distance. Thus an irregular dynamic max-min problem of extremely large-scale is formulated, where the time instant when the optimal solution can be attained is uncertain and the optimum solution depends on all the intermediate guidance commands generated before. For solving this problem, a two-step strategy is conceived. In the first step, we use the proximal policy optimization (PPO) algorithm to generate the guidance commands of the EFV. The results obtained by PPO in the global search space are coarse, despite the fact that the reward function, the neural network parameters and the learning rate are designed elaborately. Therefore, in the second step, we propose to invoke the evolution strategy (ES) based algorithm, which uses the result of PPO as the initial value, to further improve the quality of the solution by searching in the local space. Simulation results demonstrate that the proposed guidance design method based on the PPO algorithm is capable of achieving a residual velocity of 67.24 m/s, higher than the residual velocities achieved by the benchmark soft actor-critic and deep deterministic policy gradient algorithms. Furthermore, the proposed ES-enhanced PPO algorithm outperforms the PPO algorithm by 2.7\%, achieving a residual velocity of 69.04 m/s. |
| Author | Yang, Shaoshi Wang, Tianshu Gong, Min Hu, Xiao |
| Author_xml | – sequence: 1 givenname: Xiao surname: Hu fullname: Hu, Xiao – sequence: 2 givenname: Tianshu surname: Wang fullname: Wang, Tianshu – sequence: 3 givenname: Min surname: Gong fullname: Gong, Min – sequence: 4 givenname: Shaoshi surname: Yang fullname: Yang, Shaoshi |
| BookMark | eNotzctOwzAUBFALgUQp_QB2llin2L5xHktU0oJUCQkK28qP69RVcEKcVPD3BMFsZjVnrsh5aAMScsPZMi2kZHeq__KnpUiZXDLIOT8jMwHAkyIV4pIsYjwyxkSWCylhRurN6K0KBukDRl8H6tqeVtGoDum68fVhoO948KZB-hZ9qGl1aptx8G2gr0OvBqy_aRUOv4KdCOzoC_owIQY_MAx0i6oP0-6aXDjVRFz895zs1tVu9ZhsnzdPq_ttoqSApCw0KMfRai1YxlA4Lk2phc2zwjpXMJtrwVPDAK0TTmYgDWhZppnRFjnCnNz-sV3ffo4Yh_2xHfswPe6BSTEFOMAPzpxbLw |
| ContentType | Paper |
| Copyright | 2024. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: 2024. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS |
| DOI | 10.48550/arxiv.2405.03711 |
| DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection ProQuest MSED ProQuest Central (Alumni) ProQuest Central UK/Ireland ProQuest Central Essentials ProQuest Central ProQuest Technology Collection ProQuest One ProQuest Central Korea SciTech Premium Collection ProQuest Engineering Collection Engineering Database Proquest Central Premium ProQuest One Academic (New) Publicly Available Content Database ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
| DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest One Academic Middle East (New) ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest Central (New) ProQuest One Academic ProQuest One Academic (New) Engineering Collection |
| DatabaseTitleList | Publicly Available Content Database |
| Database_xml | – sequence: 1 dbid: PIMPY name: Publicly Available Content Database url: http://search.proquest.com/publiccontent sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Physics |
| EISSN | 2331-8422 |
| Genre | Working Paper/Pre-Print |
| GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS |
| ID | FETCH-LOGICAL-a523-98b3af1edbb2060e2f15c9b2d768dff80d7b214c03edf2f5635c3b5946cbde1e3 |
| IEDL.DBID | PIMPY |
| IngestDate | Mon Jun 30 09:13:13 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a523-98b3af1edbb2060e2f15c9b2d768dff80d7b214c03edf2f5635c3b5946cbde1e3 |
| Notes | SourceType-Working Papers-1 ObjectType-Working Paper/Pre-Print-1 content type line 50 |
| OpenAccessLink | https://www.proquest.com/publiccontent/docview/3052222313?pq-origsite=%requestingapplication% |
| PQID | 3052222313 |
| PQPubID | 2050157 |
| ParticipantIDs | proquest_journals_3052222313 |
| PublicationCentury | 2000 |
| PublicationDate | 20240504 |
| PublicationDateYYYYMMDD | 2024-05-04 |
| PublicationDate_xml | – month: 05 year: 2024 text: 20240504 day: 04 |
| PublicationDecade | 2020 |
| PublicationPlace | Ithaca |
| PublicationPlace_xml | – name: Ithaca |
| PublicationTitle | arXiv.org |
| PublicationYear | 2024 |
| Publisher | Cornell University Library, arXiv.org |
| Publisher_xml | – name: Cornell University Library, arXiv.org |
| SSID | ssj0002672553 |
| Score | 1.8689551 |
| SecondaryResourceType | preprint |
| Snippet | Guidance commands of flight vehicles are a series of data sets with fixed time intervals, thus guidance design constitutes a sequential decision problem and... |
| SourceID | proquest |
| SourceType | Aggregation Database |
| SubjectTerms | Algorithms Commands Deep learning Evolution Flight Flight vehicles Machine learning Neural networks Optimization Proportional navigation |
| Title | Guidance Design for Escape Flight Vehicle Using Evolution Strategy Enhanced Deep Reinforcement Learning |
| URI | https://www.proquest.com/docview/3052222313 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV07T8MwELagBYmJt3iUygOr28RO0mRCAlJgoIpKhcpU-XGhlVBakraCf4_tujAgMTFbsiz7fHff-fN3CF0qiJTOGhThOQAJeOwRAZQRCTQRnTBiIKRtNtHp9eLhMMnc9-jK0SrXPtE66pXas-FtayfcVlNpKuZtbaXURDafXc3eiekhZd5aXUONTVQ3X7w1FKtnD4_Zy3fNhUYdnUGz1eOmlfJq8_JjsmzpsBa2jHid_8sl2zjT3f3fFe7plfEZlPtoA4oDtG3ZnrI6RK93i4kyx41vLYED68wVp5XhQuHum4Hr-BnGxqKwpRTgdOksFDs120-cFmNLHtBTwAz3wSqwSltsxE609fUIDbrp4OaeuI4LhGtASpJYMJ77oISgXuQBzf1QJoIqjUlUnsee6gjqB9JjoHKa65MMJRNhEkRSKPCBHaNaMS3gBGENq3IegvSTkAYhE4JD4kmNxSlX0gP_FDXWmzhyt6Ya_ezZ2d_D52iH6uTCEg-DBqrNywVcoC25nE-qsonq12kv6zcNj_Op6YzgC4M-xCk |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1LTwIxEJ6gaPTkOz5Qe9Dj4rbdXejBeBBQIxKixHgjfcwiiUFkFfVH-R9ty6IHE28ePDdpujuvb6ZfZwAODCbGogYTyBQxiGQ1DBQyHmhkQlXihKPSfthEpdWq3t2JdgE-pm9hHK1y6hO9ozaP2tXIj6xeMhfLKD8ZPgVuapS7XZ2O0JioxSW-v9qULTu-qFn5HjLWqHdOz4N8qkAgbdIViKriMqVolGJhEiJLaayFYsbibpOm1dBUFKORDjmalKX2tLHmKhZRopVBitxuOwNFiyKo8EzBm6-SDksqFqDzyd2p7xR2JEdv_XHZRs247Hrj0R8e34exxtI_-wHLUGzLIY5WoICDVZj3bFWdrUHv7KVvnLqSmiegEIu8ST1zXC7SeHDlBnKL984iiKdEkPo4tzCSd-N9J_XBvSc_2C1wSK7Rd5DVvlhK8qazvXXo_MWnbcDs4HGAm0BsWpjKGDUVMYtirpREEeooEUwaHSLdgtJUSt3c6rPut4i2f1_eh4XzzlWz27xoXe7AIrNAyZMooxLMPo9ecBfm9Pi5n432vIYR6P6xQD8BPw8Srg |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Guidance+Design+for+Escape+Flight+Vehicle+Using+Evolution+Strategy+Enhanced+Deep+Reinforcement+Learning&rft.jtitle=arXiv.org&rft.au=Hu%2C+Xiao&rft.au=Wang%2C+Tianshu&rft.au=Gong%2C+Min&rft.au=Yang%2C+Shaoshi&rft.date=2024-05-04&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.2405.03711 |