One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering

Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appea...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE robotics and automation letters Ročník 10; číslo 6; s. 6320 - 6327
Hlavní autori: Zhu, Yifan, Xiang, Tianyi, Dollar, Aaron M., Pan, Zherong
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Piscataway IEEE 01.06.2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Computational modeling
Diff erentiable simulation and rendering
Geometry
Neural radiance field
Object recognition
Optimization
Physical properties
physics-based modeling
Point cloud compression
Rendering
Rendering (computer graphics)
Representations
Robot learning
Robots
scene understanding
Task planning (robotics)
Three-dimensional displays
Visual observation
world modeling
ISSN:2377-3766, 2377-3766
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models or rigid objects, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready rigid world models from only one robot action sequence.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2025.3566623