A Multimodal Dynamic Hand Gesture Recognition Based on Radar-Vision Fusion

Regarding increasingly complex scenarios in hand gesture recognition (HGR), it is challenging to implement a reliable HGR due to the nonadaptability of individual sensors to the environment and the discrepancy of personal habits. Multisensor fusion has been deemed an effective way to overcome the li...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	IEEE transactions on instrumentation and measurement Jg. 72; S. 1 - 15
Hauptverfasser:	Liu, Haoming, Liu, Zhenyu
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	New York IEEE 2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:	Ablation Cameras Convolution Deep learning Deformation Feature extraction Field of view Formability frequency-modulated continuous-wave (FMCW) Gesture recognition hand gesture recognition (HGR) Hidden Markov models Matching millimeter-wave (MMW) multimodal fusion Multisensor fusion Radar Reliability Sensors Time synchronization
ISSN:	0018-9456, 1557-9662
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	Regarding increasingly complex scenarios in hand gesture recognition (HGR), it is challenging to implement a reliable HGR due to the nonadaptability of individual sensors to the environment and the discrepancy of personal habits. Multisensor fusion has been deemed an effective way to overcome the limitations of a single sensor. However, there is a lack of research on HGR to effectively establish bridges linking multimodal heterogeneous information. To address this issue, we propose a novel multimodal dynamic HGR method based on a two-branch fusion deformable network with Gram matching. First, a time-synchronized method is designed to preprocess the multimodal data. Second, a two-branch network is proposed to implement gesture classification based on radar-vision fusion. The input convolution is replaced by the deformable convolution to improve the generalization of gesture motion modeling. The long short-term memory (LSTM) unit is used to extract the temporal features of dynamic hand gestures. Third, Gram matching is presented as a loss function to mine high-dimensional heterogeneous information and maintain the integrity of radar-vision fusion. The experimental results indicate that the proposed method effectively improves the adaptability of the classifier to complex environments and exhibits satisfactory robustness to multiple subjects. Furthermore, ablation analysis shows that deformable convolution and Gram loss not only provide reliable gesture recognition but also enhance the generalization ability of the proposed methods in different field-of-view scenarios.
AbstractList	Regarding increasingly complex scenarios in hand gesture recognition (HGR), it is challenging to implement a reliable HGR due to the nonadaptability of individual sensors to the environment and the discrepancy of personal habits. Multisensor fusion has been deemed an effective way to overcome the limitations of a single sensor. However, there is a lack of research on HGR to effectively establish bridges linking multimodal heterogeneous information. To address this issue, we propose a novel multimodal dynamic HGR method based on a two-branch fusion deformable network with Gram matching. First, a time-synchronized method is designed to preprocess the multimodal data. Second, a two-branch network is proposed to implement gesture classification based on radar–vision fusion. The input convolution is replaced by the deformable convolution to improve the generalization of gesture motion modeling. The long short-term memory (LSTM) unit is used to extract the temporal features of dynamic hand gestures. Third, Gram matching is presented as a loss function to mine high-dimensional heterogeneous information and maintain the integrity of radar–vision fusion. The experimental results indicate that the proposed method effectively improves the adaptability of the classifier to complex environments and exhibits satisfactory robustness to multiple subjects. Furthermore, ablation analysis shows that deformable convolution and Gram loss not only provide reliable gesture recognition but also enhance the generalization ability of the proposed methods in different field-of-view scenarios.
Author	Liu, Zhenyu Liu, Haoming
Author_xml	– sequence: 1 givenname: Haoming orcidid: 0000-0002-6467-9599 surname: Liu fullname: Liu, Haoming email: liuhaomingdut@gmail.com organization: School of Information Engineering, Guangdong University of Technology, Guangzhou, China – sequence: 2 givenname: Zhenyu orcidid: 0000-0002-4205-6067 surname: Liu fullname: Liu, Zhenyu email: zhenyuliu@gdut.edu.cn organization: School of Information Engineering, Guangdong University of Technology, Guangzhou, China
BookMark	eNp9UD1PwzAUtFCRaAs7A0Mk5hTbib_GUugHaoVUFVbLiW3kKk2KnQz99zhqB8TAdKenu_fu3QgM6qY2ANwjOEEIiqfdajPBEGeTDJNMQHoFhogQlgpK8QAMIUQ8FTmhN2AUwh5CyGjOhuBtmmy6qnWHRqsqeTnV6uDKZKlqnSxMaDtvkq0pm6_ata6pk2cVjE4i2SqtfPrpQj-ddz3cgmurqmDuLjgGH_PX3WyZrt8Xq9l0nZZY4DZlRUELQbktGDc8EkEJJoYYy22pubY6phTY0lxrUlJeiJxBSCAzmGOb42wMHs97j7757mJIuW86X8eTEjMuMkoYRVFFz6rSNyF4Y2XpWtU_0XrlKomg7HuTsTfZ9yYvvUUj_GM8endQ_vSf5eFsccaYX3JIc8RR9gP5PXkZ
CODEN	IEIMAO
CitedBy_id	crossref_primary_10_1109_JSEN_2023_3276798 crossref_primary_10_1109_TCYB_2025_3525652 crossref_primary_10_1109_ACCESS_2025_3529025 crossref_primary_10_1109_TIM_2024_3412196 crossref_primary_10_1109_TIM_2025_3582311 crossref_primary_10_1007_s11760_024_03664_6 crossref_primary_10_1109_JIOT_2025_3585249 crossref_primary_10_1109_JSEN_2024_3481964 crossref_primary_10_1007_s13735_025_00363_x crossref_primary_10_1109_TCE_2025_3557084 crossref_primary_10_1109_TIM_2023_3307768 crossref_primary_10_1109_ACCESS_2024_3456436
Cites_doi	10.1109/TSMCC.2007.893280 10.1145/2897824.2925953 10.1109/TGRS.2021.3093334 10.1109/JSEN.2022.3181518 10.1109/TIM.2021.3109732 10.1145/2157689.2157743 10.1109/LGRS.2021.3086136 10.1109/ACCESS.2017.2684186 10.1109/34.799904 10.1109/MWSCAS.2014.6908353 10.1109/CVPR.1993.341109 10.1109/TIM.2019.2909249 10.1109/TIM.2015.2498560 10.1109/THMS.2020.3036637 10.1109/LRA.2021.3089999 10.1109/34.735811 10.1109/TCSVT.2015.2469551 10.1109/MIM.2020.9082794 10.1109/JERM.2019.2949456 10.1109/ICIT46573.2021.9453703 10.1109/JSEN.2020.2994292 10.1109/THMS.2021.3131675 10.1109/MIM.2014.6825388 10.1109/TIE.2003.814758 10.1109/JSEN.2020.3014276 10.3390/rs13061064 10.1109/ICISET54810.2022.9775918 10.1109/FG.2015.7163132 10.1109/TGRS.2020.3010880 10.1201/9781420064995-c34 10.3389/fpubh.2020.582191 10.1109/ICCV.1998.710744 10.1109/RadarConf2147009.2021.9455238 10.1109/JSEN.2018.2877978 10.1109/JSAC.2022.3155526 10.1109/LGRS.2022.3217390 10.1109/34.598226 10.1007/1-4020-4295-7_11 10.1145/2984511.2984565 10.1109/ICCVW.2017.373 10.1109/ACCESS.2020.2994281 10.1109/TIM.2022.3164162 10.1109/TMTT.2020.3031619 10.1109/TII.2011.2172450 10.1109/JIOT.2021.3072169 10.1109/CVPR.2016.90 10.1109/ROMAN.2012.6343787 10.1109/TAP.1986.1143830 10.1109/LSP.2020.3026942 10.1016/j.neucom.2022.04.035 10.1109/VNC.2018.8628409 10.1109/JRA.1987.1087109 10.1109/JSEN.2020.3034911 10.1109/MSP.2019.2903715 10.1109/TIM.2021.3132924 10.1109/TGRS.2021.3122332 10.3390/rs13183791 10.1109/MIM.2015.7066676 10.1109/ACCESS.2021.3056878 10.1109/TIP.2019.2936111 10.1109/JSEN.2022.3145844 10.1109/TIM.2022.3181305 10.1109/VTC2022-Spring54318.2022.9860976
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID	97E RIA RIE AAYXX CITATION 7SP 7U5 8FD L7M
DOI	10.1109/TIM.2023.3253906
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Xplore Digital Library CrossRef Electronics & Communications Abstracts Solid State and Superconductivity Abstracts Technology Research Database Advanced Technologies Database with Aerospace
DatabaseTitle	CrossRef Solid State and Superconductivity Abstracts Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts
DatabaseTitleList	Solid State and Superconductivity Abstracts
Database_xml	– sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Physics
EISSN	1557-9662
EndPage	15
ExternalDocumentID	10_1109_TIM_2023_3253906 10064181
Genre	orig-research
GrantInformation_xml	– fundername: Guangdong Provincial Science and Technology Plan Project grantid: 2021A0505080014 – fundername: Basic and Applied Basic Research Foundation of Guangdong Province; Guangdong Basic and Applied Basic Research Foundation grantid: 2023A1515012873 funderid: 10.13039/501100021171
GroupedDBID	-~X 0R~ 29I 4.4 5GY 5VS 6IK 85S 8WZ 97E A6W AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ H~9 IAAWW IBMZZ ICLAB IDIHD IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P RIA RIE RNS TN5 TWZ VH1 VJK AAYXX CITATION 7SP 7U5 8FD L7M
ID	FETCH-LOGICAL-c292t-7bb6b968fb78e896896525e5ef8fcd8dfd45692f64dd5c68b94700507e282f423
IEDL.DBID	RIE
ISICitedReferencesCount	29
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001012063600017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	0018-9456
IngestDate	Mon Jun 30 10:13:23 EDT 2025 Tue Nov 18 22:33:19 EST 2025 Sat Nov 29 07:58:00 EST 2025 Wed Aug 27 02:03:33 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c292t-7bb6b968fb78e896896525e5ef8fcd8dfd45692f64dd5c68b94700507e282f423
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-4205-6067 0000-0002-6467-9599
PQID	2789365761
PQPubID	85462
PageCount	15
ParticipantIDs	crossref_citationtrail_10_1109_TIM_2023_3253906 ieee_primary_10064181 proquest_journals_2789365761 crossref_primary_10_1109_TIM_2023_3253906
PublicationCentury	2000
PublicationDate	20230000 2023-00-00 20230101
PublicationDateYYYYMMDD	2023-01-01
PublicationDate_xml	– year: 2023 text: 20230000
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on instrumentation and measurement
PublicationTitleAbbrev	TIM
PublicationYear	2023
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref57 ref12 ref56 ref15 ref59 ref14 ref58 Sun (ref18) 2021 Zhang (ref39) 2020 ref53 ref52 ref11 ref10 ref54 ref17 ref16 ref19 ref51 ref50 ref46 ref45 ref48 ref47 ref42 ref41 ref44 ref43 ref49 ref8 Gatys (ref24); 28 ref7 ref9 ref4 ref3 ref6 ref5 ref40 ref35 ref34 ref37 ref36 ref31 ref30 ref33 ref32 ref2 ref1 ref38 ref23 ref67 ref26 ref25 ref20 Kulemin (ref55) 2003 ref64 ref63 ref22 ref66 ref21 ref65 ref28 ref27 ref29 ref60 ref62 ref61
References_xml	– ident: ref28 doi: 10.1109/TSMCC.2007.893280 – ident: ref19 doi: 10.1145/2897824.2925953 – ident: ref26 doi: 10.1109/TGRS.2021.3093334 – ident: ref41 doi: 10.1109/JSEN.2022.3181518 – ident: ref11 doi: 10.1109/TIM.2021.3109732 – ident: ref42 doi: 10.1145/2157689.2157743 – ident: ref44 doi: 10.1109/LGRS.2021.3086136 – ident: ref22 doi: 10.1109/ACCESS.2017.2684186 – ident: ref21 doi: 10.1109/34.799904 – ident: ref53 doi: 10.1109/MWSCAS.2014.6908353 – ident: ref34 doi: 10.1109/CVPR.1993.341109 – ident: ref49 doi: 10.1109/TIM.2019.2909249 – ident: ref15 doi: 10.1109/TIM.2015.2498560 – ident: ref48 doi: 10.1109/THMS.2020.3036637 – ident: ref8 doi: 10.1109/LRA.2021.3089999 – year: 2020 ident: ref39 article-title: MediaPipe hands: On-device real-time hand tracking publication-title: arXiv:2006.10214 – ident: ref35 doi: 10.1109/34.735811 – ident: ref13 doi: 10.1109/TCSVT.2015.2469551 – ident: ref2 doi: 10.1109/MIM.2020.9082794 – ident: ref56 doi: 10.1109/JERM.2019.2949456 – ident: ref61 doi: 10.1109/ICIT46573.2021.9453703 – ident: ref43 doi: 10.1109/JSEN.2020.2994292 – ident: ref5 doi: 10.1109/THMS.2021.3131675 – ident: ref14 doi: 10.1109/MIM.2014.6825388 – ident: ref37 doi: 10.1109/TIE.2003.814758 – ident: ref57 doi: 10.1109/JSEN.2020.3014276 – ident: ref59 doi: 10.3390/rs13061064 – ident: ref16 doi: 10.1109/ICISET54810.2022.9775918 – ident: ref62 doi: 10.1109/FG.2015.7163132 – ident: ref23 doi: 10.1109/TGRS.2020.3010880 – ident: ref33 doi: 10.1201/9781420064995-c34 – ident: ref9 doi: 10.3389/fpubh.2020.582191 – ident: ref36 doi: 10.1109/ICCV.1998.710744 – ident: ref51 doi: 10.1109/RadarConf2147009.2021.9455238 – ident: ref6 doi: 10.1109/JSEN.2018.2877978 – ident: ref12 doi: 10.1109/JSAC.2022.3155526 – ident: ref47 doi: 10.1109/LGRS.2022.3217390 – ident: ref27 doi: 10.1109/34.598226 – ident: ref17 doi: 10.1007/1-4020-4295-7_11 – ident: ref64 doi: 10.1145/2984511.2984565 – ident: ref67 doi: 10.1109/ICCVW.2017.373 – ident: ref50 doi: 10.1109/ACCESS.2020.2994281 – ident: ref10 doi: 10.1109/TIM.2022.3164162 – ident: ref29 doi: 10.1109/TMTT.2020.3031619 – ident: ref38 doi: 10.1109/TII.2011.2172450 – ident: ref46 doi: 10.1109/JIOT.2021.3072169 – ident: ref66 doi: 10.1109/CVPR.2016.90 – ident: ref40 doi: 10.1109/ROMAN.2012.6343787 – ident: ref45 doi: 10.1109/TAP.1986.1143830 – ident: ref54 doi: 10.1109/LSP.2020.3026942 – year: 2021 ident: ref18 article-title: Signal processing algorithms for gesture recognition using millimeter-wave radar technology – ident: ref65 doi: 10.1016/j.neucom.2022.04.035 – ident: ref3 doi: 10.1109/VNC.2018.8628409 – ident: ref63 doi: 10.1109/JRA.1987.1087109 – volume: 28 start-page: 1 volume-title: Proc. Adv. Neural Inf. Process. Syst. ident: ref24 article-title: Texture synthesis using convolutional neural networks – ident: ref60 doi: 10.1109/JSEN.2020.3034911 – ident: ref4 doi: 10.1109/MSP.2019.2903715 – volume-title: Millimeter-Wave Radar Targets and Clutter year: 2003 ident: ref55 – ident: ref30 doi: 10.1109/TIM.2021.3132924 – ident: ref32 doi: 10.1109/TGRS.2021.3122332 – ident: ref52 doi: 10.3390/rs13183791 – ident: ref1 doi: 10.1109/MIM.2015.7066676 – ident: ref58 doi: 10.1109/ACCESS.2021.3056878 – ident: ref25 doi: 10.1109/TIP.2019.2936111 – ident: ref31 doi: 10.1109/JSEN.2022.3145844 – ident: ref7 doi: 10.1109/TIM.2022.3181305 – ident: ref20 doi: 10.1109/VTC2022-Spring54318.2022.9860976
SSID	ssj0007647
Score	2.4971404
Snippet	Regarding increasingly complex scenarios in hand gesture recognition (HGR), it is challenging to implement a reliable HGR due to the nonadaptability of...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	1
SubjectTerms	Ablation Cameras Convolution Deep learning Deformation Feature extraction Field of view Formability frequency-modulated continuous-wave (FMCW) Gesture recognition hand gesture recognition (HGR) Hidden Markov models Matching millimeter-wave (MMW) multimodal fusion Multisensor fusion Radar Reliability Sensors Time synchronization
Title	A Multimodal Dynamic Hand Gesture Recognition Based on Radar-Vision Fusion
URI	https://ieeexplore.ieee.org/document/10064181 https://www.proquest.com/docview/2789365761
Volume	72
WOSCitedRecordID	wos001012063600017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Xplore customDbUrl: eissn: 1557-9662 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0007647 issn: 0018-9456 databaseCode: RIE dateStart: 19630101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA5aFPTgoypWq-TgxcO23Udex_qoVbBIqdLbspsHCLor266_30l2Wyqi4GnnkFnCTCYzk-SbQehC0CSE3S_xIqFTLwoD6QlCAs9IIyhPmeK8ajbBRiM-nYqnGqzusDBaa_f4THcs6e7yVS5Le1QGFg4O1LdA63XGWAXWWm67jEZVgUwfLBjCgsWdZE90J_ePHdsmvBMGBHJ8-s0HuaYqP3Zi514Gu_-c2B7aqeNI3K8Uv4_WdNZE2yvVBZto073ulLMD9NDHDmj7nivguama0ONhkil8B3MpC43Hi4dEeYavwLMpDMQ4UUnhvTj4OR6U9nOInge3k-uhVzdR8GQggrnH0pSmIHaTMq45EIKSgGiiDTdScWUUyEoEhkZKEQnqERGzRWGYhmTMQLB1hBpZnuljhDURUsjED6UUkZ8kHHggvVYGbJqEkrVQdyHWWNYVxm2ji7fYZRo9EYMiYquIuFZEC10uOT6q6hp_jD20gl8ZV8m8hdoL1cW1_c1ii-8NKeRS_skvbKdoy_69Ok1po8a8KPUZ2pCf89dZce6W1hfx5ckG
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT8MwDLbQAAEHHgPEYEAOXDh0o4-0yXE8xgbbhKaBdqvaPCQk6NAe_H6ctJuGEEic6kOsRnYc20k-G-CCh4mPu1_iBFylTuB7wuGUeo4WmocsjSRjebOJqNdjwyF_KsDqFgujlLKPz1TNkPYuX47EzByVoYWjA3UN0HqVBoHn5nCtxcYbhUFeItNFG8bAYH4recXrg3a3ZhqF13yPYpYffvNCtq3Kj73YOpjmzj-ntgvbRSRJGrnq92BFZWXYWqovWIZ1-75TTPbhoUEs1PZ9JJHnNm9DT1pJJsk9zmU2VqQ_f0o0ysg1-jZJkOgnMhk7LxaATpoz8zmA5-bd4KblFG0UHOFxb-pEaRqmKHidRkwxJHhIPaqo0kwLyaSWKCvu6TCQkgpUEA8iUxYmUpiOaQy3DqGUjTJ1BERRLrhIXF8IHrhJwpAHE2yp0aqpL6IK1OdijUVRY9y0uniLba5xxWNURGwUEReKqMDlguMjr6_xx9gDI_ilcbnMK1Cdqy4uLHASG4SvH2I25R7_wnYOG61BtxN32r3HE9g0f8rPVqpQmo5n6hTWxOf0dTI-s8vsC5P3zE0
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Multimodal+Dynamic+Hand+Gesture+Recognition+Based+on+Radar%E2%80%93Vision+Fusion&rft.jtitle=IEEE+transactions+on+instrumentation+and+measurement&rft.au=Liu%2C+Haoming&rft.au=Liu%2C+Zhenyu&rft.date=2023&rft.issn=0018-9456&rft.eissn=1557-9662&rft.volume=72&rft.spage=1&rft.epage=15&rft_id=info:doi/10.1109%2FTIM.2023.3253906&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TIM_2023_3253906
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9456&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9456&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9456&client=summon