One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering

Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appea...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE robotics and automation letters Vol. 10; no. 6; pp. 6320 - 6327
Main Authors:	Zhu, Yifan, Xiang, Tianyi, Dollar, Aaron M., Pan, Zherong
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 01.06.2025 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Computational modeling Diff erentiable simulation and rendering Geometry Neural radiance field Object recognition Optimization Physical properties physics-based modeling Point cloud compression Rendering Rendering (computer graphics) Representations Robot learning Robots scene understanding Task planning (robotics) Three-dimensional displays Visual observation world modeling
ISSN:	2377-3766, 2377-3766
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models or rigid objects, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready rigid world models from only one robot action sequence.
AbstractList	Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models or rigid objects, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready rigid world models from only one robot action sequence.
Author	Xiang, Tianyi Pan, Zherong Zhu, Yifan Dollar, Aaron M.
Author_xml	– sequence: 1 givenname: Yifan orcidid: 0000-0002-4587-4305 surname: Zhu fullname: Zhu, Yifan email: yifan.zhu@yale.edu organization: Department of Mechanical Engineering and Materials Science, Yale University, New Haven, CT, USA – sequence: 2 givenname: Tianyi surname: Xiang fullname: Xiang, Tianyi email: tianyi.xiang@yale.edu organization: Department of Mechanical Engineering and Materials Science, Yale University, New Haven, CT, USA – sequence: 3 givenname: Aaron M. orcidid: 0000-0002-2409-4668 surname: Dollar fullname: Dollar, Aaron M. email: aaron.dollar@yale.edu organization: Department of Mechanical Engineering and Materials Science, Yale University, New Haven, CT, USA – sequence: 4 givenname: Zherong orcidid: 0000-0001-9348-526X surname: Pan fullname: Pan, Zherong email: zherong.pan.usa@gmail.com organization: LightSpeed Studios, Seattle, WA, USA
BookMark	eNpNkM1PAjEQxRuDiYjcPXjYxPNiP3bb7ZEgqJGEBPTcdLtTLVla7C4m_veWwIHTm8n83szk3aKBDx4Quid4QgiWT8v1dEIxLSes5JxTdoWGlAmRM8H54KK-QeOu22KMSUkFk-UQva885Jvv0Gdr0G3eh3zjdtmv09ncN8c2SfbsrIUIvne6biFLxKHVvQs-02m6Bt9AdP7rDl1b3XYwPusIfS7mH7PXfLl6eZtNl7mhhehzI2va6JrXTV2JQjKDAURJrBRAmNbUAEDV1KAJFNhUDZZVZTgnzBTC2MayEXo87d3H8HOArlfbcIg-nVSMYiZFWZEiUfhEmRi6LoJV--h2Ov4pgtUxNZVSU8fU1Dm1ZHk4WVx64QKXFSWYsn9yGGn1
CODEN	IRALC6
Cites_doi	10.1145/3592433 10.1002/(SICI)1097-0207(19960815)39:15<2673::AID-NME972>3.0.CO;2-I 10.1109/CVPR52733.2024.00420 10.1109/LRA.2021.3062323 10.1145/3414685.3417766 10.1016/0005-1098(71)90059-8 10.15607/RSS.2021.XVII.008 10.15607/RSS.2020.XVI.099 10.1109/robot.2003.1242171 10.1145/3414685.3417861 10.15607/RSS.2021.XVII.034 10.1145/3528223.3530155 10.1145/3503250 10.1109/ICRA.2015.7139807 10.1023/A:1008292328909 10.1109/3DV53792.2021.00020 10.1109/CVPR42600.2020.00011 10.1109/ICCV.2019.00780 10.1109/CVPR.2018.00308 10.1109/CVPR42600.2020.00749 10.1109/MRA.2015.2448951 10.1145/882262.882358 10.1109/CVPR42600.2020.00030
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/LRA.2025.3566623
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	2377-3766
EndPage	6327
ExternalDocumentID	10_1109_LRA_2025_3566623 10982102
Genre	orig-research
GroupedDBID	0R~ 97E AAJGR AASAJ AAWTH ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c247t-c9b2dab6bdb87493c0ee751f97e13aa2ceee8dbea1e40c8d0988c6613c47cfdf3
IEDL.DBID	RIE
ISICitedReferencesCount	0
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001501968400002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2377-3766
IngestDate	Sat Nov 22 13:41:14 EST 2025 Thu Nov 27 01:01:19 EST 2025 Wed Nov 26 07:22:47 EST 2025
IsPeerReviewed	true
IsScholarly	true
Issue	6
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c247t-c9b2dab6bdb87493c0ee751f97e13aa2ceee8dbea1e40c8d0988c6613c47cfdf3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-2409-4668 0000-0002-4587-4305 0000-0001-9348-526X
PQID	3203975814
PQPubID	4437225
PageCount	8
ParticipantIDs	crossref_primary_10_1109_LRA_2025_3566623 ieee_primary_10982102 proquest_journals_3203975814
PublicationCentury	2000
PublicationDate	2025-06-01
PublicationDateYYYYMMDD	2025-06-01
PublicationDate_xml	– month: 06 year: 2025 text: 2025-06-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE robotics and automation letters
PublicationTitleAbbrev	LRA
PublicationYear	2025
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref12 ref15 ref14 ref31 Tochilkin (ref35) 2024 ref30 ref33 Zhou (ref36) 2018 ref10 ref32 Peng (ref11) 2021; 34 Qiao (ref16) 2021 ref38 Yang (ref4) 2024 Ha (ref1) 2018; 31 ref18 strm (ref24) 1971; 7 Hong (ref34) 2024 Kazhdan (ref26) 2006; 7 Howell (ref17) 2022 ref23 Coumans (ref37) 2016 ref20 Bruce (ref2) 2024 ref22 Zhu (ref3) 2024 ref28 ref27 ref8 ref7 ref9 Ravi (ref29) 2020 Abou-Chakra (ref25) 2024 Macklin (ref39) 2024; 43 ref6 ref5 ref40 Achlioptas (ref19) 2018 Pfrommer (ref21) 2021
References_xml	– ident: ref9 doi: 10.1145/3592433 – ident: ref33 doi: 10.1002/(SICI)1097-0207(19960815)39:15<2673::AID-NME972>3.0.CO;2-I – ident: ref23 doi: 10.1109/CVPR52733.2024.00420 – volume-title: Proc. Int. Conf. Learn. Representations year: 2024 ident: ref4 article-title: Learning interactive real-world simulators – start-page: 40 volume-title: Proc. Int. Conf. Mach. Learn. year: 2018 ident: ref19 article-title: Learning representations and generative models for 3D point clouds – year: 2022 ident: ref17 article-title: Dojo: A differentiable physics engine for robotics – volume-title: Proc. 8th Annu. Conf. Robot Learn. year: 2024 ident: ref25 article-title: Physically embodied Gaussian splatting: A visually learnt and physically grounded 3D representation for robotics – year: 2016 ident: ref37 article-title: Pybullet, a python module for physics simulation for games, robotics and machine learning – ident: ref6 doi: 10.1109/LRA.2021.3062323 – ident: ref18 doi: 10.1145/3414685.3417766 – volume: 7 start-page: 123 issue: 2 year: 1971 ident: ref24 article-title: System identificationa survey publication-title: Automatica doi: 10.1016/0005-1098(71)90059-8 – ident: ref15 doi: 10.15607/RSS.2021.XVII.008 – start-page: 2279 volume-title: Proc. Conf. Robot Learn. year: 2021 ident: ref21 article-title: ContactNets: Learning discontinuous contact dynamics with smooth, implicit representations – ident: ref5 doi: 10.15607/RSS.2020.XVI.099 – ident: ref32 doi: 10.1109/robot.2003.1242171 – volume: 34 start-page: 13032 year: 2021 ident: ref11 article-title: Shape as points: A differentiable Poisson solver publication-title: Adv. Neural Inf. Process. Syst. – ident: ref28 doi: 10.1145/3414685.3417861 – ident: ref14 doi: 10.15607/RSS.2021.XVII.034 – ident: ref22 doi: 10.1145/3528223.3530155 – volume: 31 volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2018 ident: ref1 article-title: Recurrent world models facilitate policy evolution – ident: ref8 doi: 10.1145/3503250 – start-page: 17123 volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2021 ident: ref16 article-title: Differentiable simulation of soft multi-body systems – year: 2018 ident: ref36 article-title: Open3D: A modern library for 3D data processing – ident: ref10 doi: 10.1109/ICRA.2015.7139807 – ident: ref31 doi: 10.1023/A:1008292328909 – ident: ref13 doi: 10.1109/3DV53792.2021.00020 – ident: ref20 doi: 10.1109/CVPR42600.2020.00011 – ident: ref27 doi: 10.1109/ICCV.2019.00780 – year: 2020 ident: ref29 article-title: Accelerating 3D deep learning with PyTorch3D – year: 2024 ident: ref2 article-title: Genie: Generative Interactive Environments – ident: ref12 doi: 10.1109/CVPR.2018.00308 – ident: ref30 doi: 10.1109/CVPR42600.2020.00749 – ident: ref38 doi: 10.1109/MRA.2015.2448951 – year: 2024 ident: ref35 article-title: TripoSR: Fast 3D object reconstruction from a single image – volume-title: Proc. Int. Conf. Learn. Representations year: 2024 ident: ref34 article-title: LRM: Large reconstruction model for single image to 3D – volume: 43 issue: 1 volume-title: Proc. ACM Trans. Graph. year: 2024 ident: ref39 article-title: Warp: Diff erentiable spatial computing for python – year: 2024 ident: ref3 article-title: IRASim: Learning interactive real-robot action simulators – ident: ref40 doi: 10.1145/882262.882358 – ident: ref7 doi: 10.1109/CVPR42600.2020.00030 – volume: 7 start-page: 61 issue: 4 volume-title: Proc. 4th Eurograph. Symp. Geometry Process. year: 2006 ident: ref26 article-title: Poisson surface reconstruction
SSID	ssj0001527395
Score	2.2930155
Snippet	Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments....
SourceID	proquest crossref ieee
SourceType	Aggregation Database Index Database Publisher
StartPage	6320
SubjectTerms	Computational modeling Diff erentiable simulation and rendering Geometry Neural radiance field Object recognition Optimization Physical properties physics-based modeling Point cloud compression Rendering Rendering (computer graphics) Representations Robot learning Robots scene understanding Task planning (robotics) Three-dimensional displays Visual observation world modeling
Title	One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering
URI	https://ieeexplore.ieee.org/document/10982102 https://www.proquest.com/docview/3203975814
Volume	10
WOSCitedRecordID	wos001501968400002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT9tAEB0RxKE9tBSomjZEPnDhsKm9_ljvMYJElZoCSgDlZnlnxyISOBUYjv3tnV07AoR66Mm2vB6tnz07b3Z3ZgCOsEJEGRmhYpOJJKNE6KrkwVBSlVpih8Pn6b6eqbOzfLnUF12wuo-FISK_-YxG7tSv5ds1PrqpMtZwnTsXpQc9pbI2WOt5QsWlEtPpZiky1N9n8zE7gDIdxcxZMhm_Mj2-lsqbAdhblenH_-zPLnzo6GMwbr_3J9iieg_ev0gquA8_z2sSi5t1E8yZBIpmLRaru-BpVQaT2rpLPgSnXV0U1m9zSwG36Mp4BSXfnfv6ciztAK6mk8uTH6IrmSBQJqoRqI20pcmMNblKdIwhkUqjSiuK4rKUbBIpt4bKiJIQc8svkCOb6BgThZWt4s-wXa9r-gKB0lUUUsWet80ScvJSg449WUSyEvtwvEGz-N1mxii8RxHqgpEvHPJFh3wfDhx6L9q1wPVhsMG_6HTnoYhlyCQpzaPk6z8e-wbvnPR2x9YAtpv7RzqEHXxqVg_3Q-j9-jMZ-p_jL7S1uaA
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fb9MwED5BmQQ8AIMhCgPysJc9uCSO88OPFWwqailT26G-RfH5olWCFK1p_37OTqp1QjzwlER2nPhLzved7bsDOMMKEWVkRBabVKiUlNBVyYOhpCqxxAaHj9P9Y5JNp_lyqa86Z3XvC0NEfvMZDdypX8u3a9y6qTKWcJ07E-UhPEqUkmHrrnU3peKCielkvxgZ6k-T2ZBNQJkMYmYtqYzvKR-fTeWvIdjrlcvn__lGL-BZRyCDYfvFj-EB1S_h6UFYwVcw_l6TmN-sm2DGNFA0azFf_Qp2qzK4qK275EPwpcuMwhJuflLANbpEXkHJpTOfYY5bO4Hry4vF55HokiYIlCprBGojbWlSY02eKR1jSJQlUaUziuKylKwUKbeGyohUiLnlDuTISjpGlWFlq_g19Op1TW8gyHQVhVSx7W1TRa69xKDjTxaRrMQ-nO_RLH63sTEKb1OEumDkC4d80SHfhxOH3kG9Frg-nO7xLzrp2RSxDJkmJXmk3v7jto_weLT4NikmX6fjd_DEPandv3UKveZ2S-_hCHfNanP7wf8ifwAhNLu2
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=One-Shot+Real-to-Sim+via+End-to-End+Differentiable+Simulation+and+Rendering&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Zhu%2C+Yifan&rft.au=Xiang%2C+Tianyi&rft.au=Dollar%2C+Aaron+M.&rft.au=Pan%2C+Zherong&rft.date=2025-06-01&rft.pub=IEEE&rft.eissn=2377-3766&rft.volume=10&rft.issue=6&rft.spage=6320&rft.epage=6327&rft_id=info:doi/10.1109%2FLRA.2025.3566623&rft.externalDocID=10982102
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon