CM-AVAE: Cross-Modal Adversarial Variational Autoencoder for Visual-to-Tactile Data Generation

Vibration acceleration signals allow humans to perceive the surface characteristics of textures during tool-surface interactions. However, acquiring acceleration signals requires a specialized system, which is relatively expensive. Conversely, visual images are more accessible than acceleration sign...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE robotics and automation letters Vol. 9; no. 6; pp. 5214 - 5221
Main Authors:	Xi, Qiyuan, Wang, Fei, Tao, Liangze, Zhang, Hanjing, Jiang, Xun, Wu, Juan
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 01.06.2024 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Ablation cross-modal learning Data models Discriminators Generative adversarial networks generative adversarial networks (GANs) Generators Haptic interfaces Modal data Signal generation Solid modeling Surface properties variational autoencoders (VAEs) Vectors Vibration Vibration acceleration signals Vibration perception Vibrations Visual signals Visualization
ISSN:	2377-3766, 2377-3766
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Vibration acceleration signals allow humans to perceive the surface characteristics of textures during tool-surface interactions. However, acquiring acceleration signals requires a specialized system, which is relatively expensive. Conversely, visual images are more accessible than acceleration signals, and generative models can convert visual images into vibration acceleration signals. Utilizing generative models to generate vibration acceleration signals from visual data circumvents the need for time-consuming actual measurements. Furthermore, this approach can be applied to robot-related tasks. This letter presents a cross-modal adversarial variational autoencoder (CM-AVAE) for visual-to-tactile data generation. Our model incorporates latent space learning from variational autoencoders (VAEs) into generative adversarial networks (GANs) and maps the generator's decoder feature vectors to the discriminator. In addition, a public dataset is chosen to train the model, and relevant evaluation metrics are established to evaluate the model's generated results. The results generated by the CM-AVAE model show significant improvement in objective experiments compared to the baseline models. Furthermore, subjective experimental outcomes also surpass those of the baseline models. Ablation study shows that CM-AVAE introduces latent space learning and maps the decoder feature vectors in the generator to the discriminator, significantly improving the quality of cross-modal data generation.
AbstractList	Vibration acceleration signals allow humans to perceive the surface characteristics of textures during tool-surface interactions. However, acquiring acceleration signals requires a specialized system, which is relatively expensive. Conversely, visual images are more accessible than acceleration signals, and generative models can convert visual images into vibration acceleration signals. Utilizing generative models to generate vibration acceleration signals from visual data circumvents the need for time-consuming actual measurements. Furthermore, this approach can be applied to robot-related tasks. This letter presents a cross-modal adversarial variational autoencoder (CM-AVAE) for visual-to-tactile data generation. Our model incorporates latent space learning from variational autoencoders (VAEs) into generative adversarial networks (GANs) and maps the generator's decoder feature vectors to the discriminator. In addition, a public dataset is chosen to train the model, and relevant evaluation metrics are established to evaluate the model's generated results. The results generated by the CM-AVAE model show significant improvement in objective experiments compared to the baseline models. Furthermore, subjective experimental outcomes also surpass those of the baseline models. Ablation study shows that CM-AVAE introduces latent space learning and maps the decoder feature vectors in the generator to the discriminator, significantly improving the quality of cross-modal data generation.
Author	Wu, Juan Jiang, Xun Tao, Liangze Xi, Qiyuan Wang, Fei Zhang, Hanjing
Author_xml	– sequence: 1 givenname: Qiyuan orcidid: 0009-0005-1066-5596 surname: Xi fullname: Xi, Qiyuan organization: School of Instrument Science and Engineering, Southeast University, Nanjing, China – sequence: 2 givenname: Fei orcidid: 0000-0003-3211-1701 surname: Wang fullname: Wang, Fei organization: School of Instrument Science and Engineering, Southeast University, Nanjing, China – sequence: 3 givenname: Liangze orcidid: 0009-0001-4475-2824 surname: Tao fullname: Tao, Liangze organization: School of Instrument Science and Engineering, Southeast University, Nanjing, China – sequence: 4 givenname: Hanjing orcidid: 0009-0003-4697-2286 surname: Zhang fullname: Zhang, Hanjing organization: School of Instrument Science and Engineering, Southeast University, Nanjing, China – sequence: 5 givenname: Xun orcidid: 0009-0007-0118-3311 surname: Jiang fullname: Jiang, Xun organization: School of Instrument Science and Engineering, Southeast University, Nanjing, China – sequence: 6 givenname: Juan orcidid: 0000-0001-8886-9920 surname: Wu fullname: Wu, Juan email: juanwuseu@seu.edu.cn organization: School of Instrument Science and Engineering, Southeast University, Nanjing, China
BookMark	eNp9kEtLw0AUhQepYK3du3ARcJ0670nchVir0CJI7dIwSW5gSszUmangvzdpXRQXrs7lcr77OJdo1NkOELomeEYITu-Wr9mMYspnjCWKcHmGxpQpFTMl5eikvkBT77cYYyKoYqkYo_d8FWebbH4f5c56H69srdsoq7_Aee1MX28GCcZ2Q38fLHSVrcFFjXXRxvi9buNg47WugmkhetBBRwvowB2YK3Te6NbD9Fcn6O1xvs6f4uXL4jnPlnFFUxriCgCnpaJpKYioG0pl0iRJU6YNF6ROKo0FoXWaKJVgCUyWTIDQnCsCwHkCbIJuj3N3zn7uwYdia_euP9kXDHMuBaaY9C58dFXDrw6aYufMh3bfBcHFEGTRB1kMQRa_QfaI_INUJhxeC06b9j_w5ggaADjZw1NJlGQ_x1-AkQ
CODEN	IRALC6
CitedBy_id	crossref_primary_10_1109_TIM_2025_3604922 crossref_primary_10_1016_j_image_2025_117338 crossref_primary_10_1109_JIOT_2025_3548144
Cites_doi	10.1145/3126686.3126723 10.1016/j.cag.2021.09.007 10.1016/j.cmpb.2022.107024 10.1109/LRA.2021.3095925 10.1109/CVPR.2019.01086 10.1109/ICRA.2019.8793763 10.1109/ICRA.2017.7989116 10.1145/3422622 10.1109/TMM.2021.3119860 10.1109/ISMAR-Adjunct57072.2022.00174 10.1561/2200000056 10.1007/s12555-022-0358-3 10.1109/CVPR.2017.632 10.1109/WHC.2015.7177716 10.1007/978-3-319-24574-4_28 10.1049/ccs2.12008 10.1007/978-3-030-58539-6_43 10.1007/s10710-017-9314-z 10.1109/TMMS.1970.299963 10.1109/TIP.2022.3219228 10.1037/0096-1523.12.4.517 10.1109/TOH.2016.2625787 10.1109/ICRA48891.2023.10160373 10.1109/HAPTICS.2014.6775475 10.1109/ICCV51070.2023.00648 10.1007/978-3-319-93399-3_3 10.1109/TASSP.1984.1164317 10.1109/TASE.2020.2971713 10.1609/aaai.v34i07.6701 10.1109/ICASSP49357.2023.10094745
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/LRA.2024.3387146
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	2377-3766
EndPage	5221
ExternalDocumentID	10_1109_LRA_2024_3387146 10496176
Genre	orig-research
GrantInformation_xml	– fundername: National Natural Science Foundation of China grantid: 62073073 funderid: 10.13039/501100001809 – fundername: Natural Science Foundation of Jiangsu Province grantid: BE2022160 funderid: 10.13039/501100004608 – fundername: National Key Research and Development Program of China grantid: 2023YFC3603602
GroupedDBID	0R~ 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c292t-cee09b729b515df2268f88fb9f451d8ca0512d9877806e36b35e5a4471ee448e3
IEDL.DBID	RIE
ISICitedReferencesCount	4
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001209593700019&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2377-3766
IngestDate	Mon Jun 30 15:12:12 EDT 2025 Tue Nov 18 21:27:59 EST 2025 Sat Nov 29 06:03:30 EST 2025 Wed Aug 27 02:06:36 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	6
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c292t-cee09b729b515df2268f88fb9f451d8ca0512d9877806e36b35e5a4471ee448e3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0001-8886-9920 0009-0007-0118-3311 0009-0005-1066-5596 0000-0003-3211-1701 0009-0003-4697-2286 0009-0001-4475-2824
PQID	3044650201
PQPubID	4437225
PageCount	8
ParticipantIDs	proquest_journals_3044650201 ieee_primary_10496176 crossref_primary_10_1109_LRA_2024_3387146 crossref_citationtrail_10_1109_LRA_2024_3387146
PublicationCentury	2000
PublicationDate	2024-06-01
PublicationDateYYYYMMDD	2024-06-01
PublicationDate_xml	– month: 06 year: 2024 text: 2024-06-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE robotics and automation letters
PublicationTitleAbbrev	LRA
PublicationYear	2024
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref35 ref12 ref34 ref14 ref31 ref30 ref11 ref33 ref10 ref32 ref2 ref1 ref17 ref16 ref19 ref18 Shi (ref24) 2019 ref26 ref25 ref20 Truong (ref15) 2023 ref22 ref21 Zhong (ref4) 2023 ref28 ref27 ref8 Oord (ref23) 2016 ref7 ref9 Kingma (ref29) 2013 ref3 ref6 ref5
References_xml	– ident: ref25 doi: 10.1145/3126686.3126723 – start-page: 1618 volume-title: Proc. Conf. Robot Learn. year: 2023 ident: ref4 article-title: Touching a nerf: Leveraging neural radiance fields for tactile sensory data generation – ident: ref26 doi: 10.1016/j.cag.2021.09.007 – ident: ref32 doi: 10.1016/j.cmpb.2022.107024 – ident: ref30 doi: 10.1109/LRA.2021.3095925 – ident: ref11 doi: 10.1109/CVPR.2019.01086 – ident: ref19 doi: 10.1109/ICRA.2019.8793763 – ident: ref2 doi: 10.1109/ICRA.2017.7989116 – ident: ref12 doi: 10.1145/3422622 – ident: ref8 doi: 10.1109/TMM.2021.3119860 – ident: ref9 doi: 10.1109/ISMAR-Adjunct57072.2022.00174 – ident: ref13 doi: 10.1561/2200000056 – ident: ref3 doi: 10.1007/s12555-022-0358-3 – year: 2013 ident: ref29 article-title: Auto-encoding variational Bayes – ident: ref21 doi: 10.1109/CVPR.2017.632 – ident: ref7 doi: 10.1109/WHC.2015.7177716 – ident: ref17 doi: 10.1007/978-3-319-24574-4_28 – ident: ref31 doi: 10.1049/ccs2.12008 – ident: ref33 doi: 10.1007/978-3-030-58539-6_43 – ident: ref5 doi: 10.1007/s10710-017-9314-z – ident: ref1 doi: 10.1109/TMMS.1970.299963 – ident: ref16 doi: 10.1109/TIP.2022.3219228 – ident: ref22 doi: 10.1037/0096-1523.12.4.517 – ident: ref18 doi: 10.1109/TOH.2016.2625787 – ident: ref28 doi: 10.1109/ICRA48891.2023.10160373 – ident: ref6 doi: 10.1109/HAPTICS.2014.6775475 – ident: ref27 doi: 10.1109/ICCV51070.2023.00648 – ident: ref10 doi: 10.1007/978-3-319-93399-3_3 – ident: ref35 doi: 10.1109/TASSP.1984.1164317 – ident: ref20 doi: 10.1109/TASE.2020.2971713 – ident: ref34 doi: 10.1609/aaai.v34i07.6701 – year: 2023 ident: ref15 article-title: Text-guided real-world-to-3D generative models with real-time rendering on mobile devices publication-title: Authorea Preprints – year: 2016 ident: ref23 article-title: WaveNet: A generative model for raw audio – volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2019 ident: ref24 article-title: Variational mixture-of-experts autoencoders for multi-modal deep generative models – ident: ref14 doi: 10.1109/ICASSP49357.2023.10094745
SSID	ssj0001527395
Score	2.285171
Snippet	Vibration acceleration signals allow humans to perceive the surface characteristics of textures during tool-surface interactions. However, acquiring...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	5214
SubjectTerms	Ablation cross-modal learning Data models Discriminators Generative adversarial networks generative adversarial networks (GANs) Generators Haptic interfaces Modal data Signal generation Solid modeling Surface properties variational autoencoders (VAEs) Vectors Vibration Vibration acceleration signals Vibration perception Vibrations Visual signals Visualization
Title	CM-AVAE: Cross-Modal Adversarial Variational Autoencoder for Visual-to-Tactile Data Generation
URI	https://ieeexplore.ieee.org/document/10496176 https://www.proquest.com/docview/3044650201
Volume	9
WOSCitedRecordID	wos001209593700019&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED7RigEG3ojykgcWBpe0zsNmi0oRA0UIQdSJyLEdqVLVoDZh5Lfjc1KohEBiSgafZfls33c-33cAF1LwMPS0pJHII-orJuw5mPlUM59pxWWQuyzX5D56eODjsXhsktVdLowxxj0-M138dbF8XagKr8rsDveFtbhhC1pRFNbJWt8XKkglJoJlKNITV_dPsXUA-37XumGRg7grpsfVUvlxADurcrv9z_HswFYDH0lc63sX1sxsDzZXSAX34XUwonESD6_JAIdBR4VGCSy8vJC43EiCn_oOkMRVWSCXpTZzYvErSSaLSk5pWdBnTHmYGnIjS0lqdmqUOYCX2-Hz4I42VRSo6ot-Sa0V9ERmMXRmoYvOLdziOed5JnI_6GmupN2WfS14FHEvNCzMWGAC6VujZYz13Qw7hPasmJkjICLIlGJM-8b6hYHIJLIN5hlWU-daat6Bq-UEp6qhGMdKF9PUuRqeSK1KUlRJ2qikA5dfEm81vcYfbQ9QBSvt6tnvwOlSiWmzARcpw0B1YLFw7_gXsRPYwN7rZ1-n0C7nlTmDdfVeThbzc2iNPobnboV9AuYUzS8
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT-MwEB0tLBJwYFk-RIFlfdjLHgwhthObW1RAoG0rtCoVJyLHdiSkqkFtyu_H46RLpRVInJKDR7Y8tueNx_MG4JdWMkkiq2mqypRyw5Q_BwtOLePMGqlFGbJcR710MJAPD-quTVYPuTDOufD4zJ3ib4jl28rM8arM73CuvMVNVuCr4DyOmnSttysVJBNTYhGMjNRZ72_mXcCYn3pHLA0gd8n4hGoq_x3Bwa5cf_vkiLZhqwWQJGs0_h2-uMkObC7RCu7CY7dPs1F2dUG6OAzaryxKYOnlmcYFR0b4aW4BSTavK2SztG5KPIIlo6fZXI9pXdEhJj2MHbnUtSYNPzXK7MH99dWwe0PbOgrUxCquqbeDkSo8ii48eLGlB1yylLIsVMnFuZVG-40ZWyXTVEaJY0nBhBOae7PlnPfeHNuH1Uk1cQdAlCiMYcxy5z1DoQqNfINlgfXUpdVWduBsMcG5aUnGsdbFOA_ORqRyr5IcVZK3KunA738Szw3Bxgdt91AFS-2a2e_A8UKJebsFZznDULXwaPj88B2xn7B-M-z38t7t4M8RbGBPzSOwY1itp3P3A9bMS_00m56EdfYK4BjPRQ
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CM-AVAE%3A+Cross-Modal+Adversarial+Variational+Autoencoder+for+Visual-to-Tactile+Data+Generation&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Xi%2C+Qiyuan&rft.au=Wang%2C+Fei&rft.au=Tao%2C+Liangze&rft.au=Zhang%2C+Hanjing&rft.date=2024-06-01&rft.pub=IEEE&rft.eissn=2377-3766&rft.volume=9&rft.issue=6&rft.spage=5214&rft.epage=5221&rft_id=info:doi/10.1109%2FLRA.2024.3387146&rft.externalDocID=10496176
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon