Cross-Modality Self-Attention and Fusion-Based Neural Network for Lower Limb Locomotion Mode Recognition
Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five c...
Uložené v:
| Vydané v: | IEEE transactions on automation science and engineering Ročník 22; s. 5411 - 5424 |
|---|---|
| Hlavní autori: | , , , , , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
2025
|
| Predmet: | |
| ISSN: | 1545-5955, 1558-3783 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five common lower limb locomotion modes including level walking, ramp ascent, ramp descent, stair ascent, and stair descent. First, the encoder-decoder structure is employed to enrich the channel diversity for the separation of the useful patterns from combined patterns. Second, a self-attention based cross-modality interaction module is proposed, which enables bilateral information flow between two encoding paths to fully exploit the interdependencies and to find complementary information between modalities. Third, a multi-modality fusion module is designed where the complementary features are fused by a channel-wise weighted summation whose coefficients are learned end-to-end. A benchmark dataset is collected from 10 health subjects containing EMG and IMU signals and five locomotion modes. Extensive experiments are conducted on one publicly available dataset ENABL3S and one self-collected dataset. The results show that the proposed method outperforms the compared methods with higher classification accuracy. The proposed method achieves a classification accuracy of 98.25% on ENABL3S dataset and 95.51% on the self-collected dataset. Note to Practitioners-This article aims to solve the real challenges encountered when intelligent recognition algorithms are applied in wearable robots: how to effectively and efficiently fuse the multi-modality data for better decision-making. First, most existing methods directly concatenate the multi-modality data, which increases the data dimensionality and brings computational burden. Second, existing recognition neural networks continuously compress the feature size such that the discriminative patterns are submerged in the noise and thus difficult to be identified. This research decomposes the mixed input signals on the channel dimension such that the useful patterns can be separated. Moreover, this research employs self-attention mechanism to associate correlations between two modalities and use this correlation as a new feature for subsequent representation learning, generating new, compact, and complementary features for classification. We demonstrate that the proposed network achieves 98.25% accuracy and 3.5 ms prediction time. We anticipate that the proposed network could be a general scientific and practical methodology of multi-modality signal fusion and feature learning for intelligent systems. |
|---|---|
| AbstractList | Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five common lower limb locomotion modes including level walking, ramp ascent, ramp descent, stair ascent, and stair descent. First, the encoder-decoder structure is employed to enrich the channel diversity for the separation of the useful patterns from combined patterns. Second, a self-attention based cross-modality interaction module is proposed, which enables bilateral information flow between two encoding paths to fully exploit the interdependencies and to find complementary information between modalities. Third, a multi-modality fusion module is designed where the complementary features are fused by a channel-wise weighted summation whose coefficients are learned end-to-end. A benchmark dataset is collected from 10 health subjects containing EMG and IMU signals and five locomotion modes. Extensive experiments are conducted on one publicly available dataset ENABL3S and one self-collected dataset. The results show that the proposed method outperforms the compared methods with higher classification accuracy. The proposed method achieves a classification accuracy of 98.25% on ENABL3S dataset and 95.51% on the self-collected dataset. Note to Practitioners-This article aims to solve the real challenges encountered when intelligent recognition algorithms are applied in wearable robots: how to effectively and efficiently fuse the multi-modality data for better decision-making. First, most existing methods directly concatenate the multi-modality data, which increases the data dimensionality and brings computational burden. Second, existing recognition neural networks continuously compress the feature size such that the discriminative patterns are submerged in the noise and thus difficult to be identified. This research decomposes the mixed input signals on the channel dimension such that the useful patterns can be separated. Moreover, this research employs self-attention mechanism to associate correlations between two modalities and use this correlation as a new feature for subsequent representation learning, generating new, compact, and complementary features for classification. We demonstrate that the proposed network achieves 98.25% accuracy and 3.5 ms prediction time. We anticipate that the proposed network could be a general scientific and practical methodology of multi-modality signal fusion and feature learning for intelligent systems. |
| Author | Song, Wenbo Zheng, Hao Pei, Zhongcai Liu, Kai Zhao, Changchen Chen, Weihai |
| Author_xml | – sequence: 1 givenname: Changchen orcidid: 0000-0002-0546-6016 surname: Zhao fullname: Zhao, Changchen organization: School of Computer Science, Hangzhou Dianzi University, Hangzhou, China – sequence: 2 givenname: Kai surname: Liu fullname: Liu, Kai organization: College of Information Engineering, Zhejiang University of Technology, Hangzhou, China – sequence: 3 givenname: Hao surname: Zheng fullname: Zheng, Hao organization: Hangzhou Innovation Institute, Beihang University, Hangzhou, China – sequence: 4 givenname: Wenbo surname: Song fullname: Song, Wenbo organization: College of Physical Education, Jilin Normal University, Siping, China – sequence: 5 givenname: Zhongcai orcidid: 0000-0001-7748-8591 surname: Pei fullname: Pei, Zhongcai organization: Hangzhou Innovation Institute, Beihang University, Hangzhou, China – sequence: 6 givenname: Weihai orcidid: 0000-0001-7912-4505 surname: Chen fullname: Chen, Weihai email: whchen@buaa.edu.cn organization: School of Electrical Engineering and Automation, Anhui University, Hefei, China |
| BookMark | eNp9kE1OwzAQhS1UJNrCAZBY5AIu_onjeFmqtiAVkGhZR44zAUMaI8dV1dvj0C4QCzYzT6P5nmbeCA1a1wJC15RMKCXqdjNdzyeMsHTCU0aZzM7QkAqRYy5zPuh1KrBQQlygUdd9kLiZKzJE7zPvug4_uko3NhySNTQ1noYAbbCuTXRbJYtdFyW-0x1UyRPsvG5iC3vnP5Pa-WTl9hCr3ZZRGrd1P2R0hOQFjHtrbT-4ROe1bjq4OvUxel3MN7N7vHpePsymK2xYlgUsyloppoDpsqw4JVABrQijHPKqlGXKFRimcqmoFpk0lOo8FVLWpGY6Ghg-RvLoa_rHPNSFsUH3FwSvbVNQUvSBFX1gRR9YcQoskvQP-eXtVvvDv8zNkbEA8GtfxKtEyr8BmJh54A |
| CODEN | ITASC7 |
| CitedBy_id | crossref_primary_10_3390_technologies13090399 crossref_primary_10_1109_LRA_2025_3597482 crossref_primary_10_1109_TII_2025_3545108 crossref_primary_10_1080_00051144_2025_2513734 |
| Cites_doi | 10.1109/EMBC.2017.8037600 10.1109/ICASSP40776.2020.9053270 10.3390/mi13081205 10.1016/j.bspc.2022.103693 10.1007/s12541-017-0081-9 10.1109/SAS48726.2020.9220037 10.1016/j.inffus.2021.11.006 10.1109/TBME.2011.2161671 10.1109/JBHI.2023.3252091 10.3389/frobt.2018.00127 10.48550/ARXIV.1706.03762 10.3389/frobt.2018.00078 10.1109/LRA.2024.3379800 10.1109/LRA.2020.3007455 10.1109/CVPRW.2018.00177 10.1109/THMS.2021.3107256 10.1109/CVPR.2019.01233 10.1038/s41598-023-28390-w 10.1109/IROS45743.2020.9341649 10.3390/s19194242 10.1109/JBHI.2023.3238406 10.1109/tnsre.2021.3087135 10.1109/tbme.2009.2034734 10.3390/s19204447 10.1109/TNSRE.2020.2987155 10.1109/JSEN.2022.3167686 10.1109/jas.2017.7510619 10.1109/tase.2023.3276856 10.1016/j.patcog.2021.108332 10.1109/TASE.2023.3250240 10.1109/JSEN.2022.3146446 10.1109/TNSRE.2022.3149793 10.1007/s11517-021-02335-9 10.1109/LRA.2021.3062003 10.1007/s42235-023-00419-w 10.1007/3-540-57868-4_57 10.1109/IROS45743.2020.9341183 10.1109/TFUZZ.2024.3364382 10.1109/RCAR54675.2022.9872282 10.1109/TBME.2008.2003293 10.1109/TBME.2017.2750139 10.1007/s11227-021-03768-7 10.1016/j.bspc.2007.07.009 10.1109/CVPR52688.2022.00520 10.1109/TNSRE.2019.2909585 10.1109/jsen.2021.3077698 10.1109/ACCESS.2020.2971552 10.1109/ICINFA.2010.5512456 10.1016/j.inffus.2022.12.023 10.1016/j.bspc.2024.106105 |
| ContentType | Journal Article |
| DBID | 97E RIA RIE AAYXX CITATION |
| DOI | 10.1109/TASE.2024.3421276 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1558-3783 |
| EndPage | 5424 |
| ExternalDocumentID | 10_1109_TASE_2024_3421276 10584554 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Natural Science Foundation of Zhejiang Province grantid: LGG21E050003 funderid: 10.13039/501100004731 – fundername: Key Research and Development Program of Zhejiang Province grantid: 2024C03042; 2021C03050 funderid: 10.13039/501100019033 |
| GroupedDBID | -~X 0R~ 29I 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AIBXA AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ H~9 IFIPE IPLJI JAVBF LAI M43 O9- OCL PQQKQ RIA RIE RNS AAYXX CITATION |
| ID | FETCH-LOGICAL-c266t-5bf9929e2abbd310ede1d0213e8db7b439ec298791a567c11a84577f0f2a266c3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 7 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001263415300001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1545-5955 |
| IngestDate | Tue Nov 18 21:37:43 EST 2025 Sat Nov 29 08:13:27 EST 2025 Wed Aug 27 01:43:35 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c266t-5bf9929e2abbd310ede1d0213e8db7b439ec298791a567c11a84577f0f2a266c3 |
| ORCID | 0000-0001-7912-4505 0000-0002-0546-6016 0000-0001-7748-8591 |
| PageCount | 14 |
| ParticipantIDs | crossref_citationtrail_10_1109_TASE_2024_3421276 crossref_primary_10_1109_TASE_2024_3421276 ieee_primary_10584554 |
| PublicationCentury | 2000 |
| PublicationDate | 20250000 2025-00-00 |
| PublicationDateYYYYMMDD | 2025-01-01 |
| PublicationDate_xml | – year: 2025 text: 20250000 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE transactions on automation science and engineering |
| PublicationTitleAbbrev | TASE |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| References | ref13 ref12 ref15 ref14 ref11 ref10 ref17 ref16 ref19 ref18 ref50 ref46 ref45 ref48 ref47 ref42 ref41 ref44 ref43 ref49 ref8 ref7 ref9 ref4 ref3 ref6 ref5 ref40 ref35 ref34 ref37 ref36 ref31 ref30 ref33 ref32 ref2 ref1 ref39 ref38 ref24 ref23 ref26 ref25 ref20 ref22 ref21 ref28 ref27 ref29 |
| References_xml | – ident: ref42 doi: 10.1109/EMBC.2017.8037600 – ident: ref14 doi: 10.1109/ICASSP40776.2020.9053270 – ident: ref38 doi: 10.3390/mi13081205 – ident: ref27 doi: 10.1016/j.bspc.2022.103693 – ident: ref1 doi: 10.1007/s12541-017-0081-9 – ident: ref25 doi: 10.1109/SAS48726.2020.9220037 – ident: ref10 doi: 10.1016/j.inffus.2021.11.006 – ident: ref28 doi: 10.1109/TBME.2011.2161671 – ident: ref47 doi: 10.1109/JBHI.2023.3252091 – ident: ref19 doi: 10.3389/frobt.2018.00127 – ident: ref35 doi: 10.48550/ARXIV.1706.03762 – ident: ref29 doi: 10.3389/frobt.2018.00078 – ident: ref40 doi: 10.1109/LRA.2024.3379800 – ident: ref20 doi: 10.1109/LRA.2020.3007455 – ident: ref44 doi: 10.1109/CVPRW.2018.00177 – ident: ref21 doi: 10.1109/THMS.2021.3107256 – ident: ref45 doi: 10.1109/CVPR.2019.01233 – ident: ref41 doi: 10.1038/s41598-023-28390-w – ident: ref48 doi: 10.1109/IROS45743.2020.9341649 – ident: ref3 doi: 10.3390/s19194242 – ident: ref5 doi: 10.1109/JBHI.2023.3238406 – ident: ref31 doi: 10.1109/tnsre.2021.3087135 – ident: ref17 doi: 10.1109/tbme.2009.2034734 – ident: ref24 doi: 10.3390/s19204447 – ident: ref11 doi: 10.1109/TNSRE.2020.2987155 – ident: ref32 doi: 10.1109/JSEN.2022.3167686 – ident: ref43 doi: 10.1109/jas.2017.7510619 – ident: ref6 doi: 10.1109/tase.2023.3276856 – ident: ref49 doi: 10.1016/j.patcog.2021.108332 – ident: ref7 doi: 10.1109/TASE.2023.3250240 – ident: ref18 doi: 10.1109/JSEN.2022.3146446 – ident: ref23 doi: 10.1109/TNSRE.2022.3149793 – ident: ref30 doi: 10.1007/s11517-021-02335-9 – ident: ref4 doi: 10.1109/LRA.2021.3062003 – ident: ref33 doi: 10.1007/s42235-023-00419-w – ident: ref39 doi: 10.1007/3-540-57868-4_57 – ident: ref8 doi: 10.1109/IROS45743.2020.9341183 – ident: ref36 doi: 10.1109/TFUZZ.2024.3364382 – ident: ref16 doi: 10.1109/RCAR54675.2022.9872282 – ident: ref12 doi: 10.1109/TBME.2008.2003293 – ident: ref22 doi: 10.1109/TBME.2017.2750139 – ident: ref26 doi: 10.1007/s11227-021-03768-7 – ident: ref9 doi: 10.1016/j.bspc.2007.07.009 – ident: ref46 doi: 10.1109/CVPR52688.2022.00520 – ident: ref2 doi: 10.1109/TNSRE.2019.2909585 – ident: ref37 doi: 10.1109/jsen.2021.3077698 – ident: ref15 doi: 10.1109/ACCESS.2020.2971552 – ident: ref13 doi: 10.1109/ICINFA.2010.5512456 – ident: ref50 doi: 10.1016/j.inffus.2022.12.023 – ident: ref34 doi: 10.1016/j.bspc.2024.106105 |
| SSID | ssj0024890 |
| Score | 2.4298325 |
| Snippet | Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still... |
| SourceID | crossref ieee |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 5411 |
| SubjectTerms | Accuracy Cross-modality interaction Electromyography Feature extraction Fuses locomotion mode recognition lower limb neural network Neural networks Representation learning self-attention Stairs |
| Title | Cross-Modality Self-Attention and Fusion-Based Neural Network for Lower Limb Locomotion Mode Recognition |
| URI | https://ieeexplore.ieee.org/document/10584554 |
| Volume | 22 |
| WOSCitedRecordID | wos001263415300001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-3783 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0024890 issn: 1545-5955 databaseCode: RIE dateStart: 20040101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELZoxQADzyLKSx6YkFych-t6LFUrhlIhWqRuUWxfRCVIUEn5_ZydFLqAxGZFjvO4RN9357v7CLkWscyQZQMDnqUsFsIybTVnBtmGMjyyXWu82IScTHrzuXqsi9V9LQwA-OQz6Lih38u3hVm5UBn-4QiXiH8N0pBSVsVaP431ej6g4igBE0qIegsz4Op21p8O0RUM407kNkBdf5ENENpQVfGgMtr_5-0ckL2aPdJ-Ze5DsgX5Ednd6Cl4TF4G7pLsobCeYdMpvGasX5ZVWiNNc0tHKxciY3cIYJa67hy45KRKB6fIYenYKafR8eJN49AUldAPdapp9Gmdb1TkLfI8Gs4G96yWU2AGUbhkQmcKyRCEqdYWWR1YCCxCfAQ9q6VGZgImVD2pglR0pQmCFJ9OyoxnYYoLmOiENPMih1NCQxXbDB1B3bUQR6BUDN1Yp5anoY24Fm3C1-83MXWvcSd58Zp4n4OrxJkkcSZJapO0yc33Ke9Vo42_JrecOTYmVpY4--X4OdkJnW6vD51ckGa5XMEl2Taf5eJjeeW_oy9AKsX2 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTsMwELXYJODAWkRZfeCE5OIkdlMfC6IqolQIitRbFNsTUakkCFK-n7EToBeQuFmR4yyT6L0Zz8wj5EyKOEOWDQx4ljIhpWXaas4Msg1leGTb1nixiXg47IzH6r4uVve1MADgk8-g5YZ-L98WZuZCZfiHI1wi_i2SZSlEGFTlWj-t9To-pOJIAZNKynoTM-DqYtR9vEZnMBStyG2Bug4jczA0p6viYaW3-c8b2iIbNX-k3crg22QB8h2yPtdVcJc8X7lLsrvCeo5NH2GasW5ZVomNNM0t7c1ckIxdIoRZ6vpz4JLDKiGcIoulA6edRgeTF41DU1RSP9TpptGHr4yjIm-Qp9716KrPakEFZhCHSyZ1ppAOQZhqbZHXgYXAIshH0LE61shNwISqE6sgle3YBEGKTxfHGc_CFBcw0R5Zyosc9gkNlbAZuoK6bUFEoJSAttCp5WloI65lk_Cv95uYutu4E72YJt7r4CpxJkmcSZLaJE1y_n3Ka9Vq46_JDWeOuYmVJQ5-OX5KVvuju0EyuBneHpK10Kn4-kDKEVkq32ZwTFbMRzl5fzvx39Qnt-TJPQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cross-Modality+Self-Attention+and+Fusion-Based+Neural+Network+for+Lower+Limb+Locomotion+Mode+Recognition&rft.jtitle=IEEE+transactions+on+automation+science+and+engineering&rft.au=Zhao%2C+Changchen&rft.au=Liu%2C+Kai&rft.au=Zheng%2C+Hao&rft.au=Song%2C+Wenbo&rft.date=2025&rft.issn=1545-5955&rft.eissn=1558-3783&rft.volume=22&rft.spage=5411&rft.epage=5424&rft_id=info:doi/10.1109%2FTASE.2024.3421276&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TASE_2024_3421276 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1545-5955&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1545-5955&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1545-5955&client=summon |