Physics-Based Simulation for Robotics Data Generation

I’m super excited to release a multi-year project we have been cooking at NVIDIA Robotics. Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new embodiment and/or scene, we'll have to recollect the dataset in this paradigm for the best perf. Key Idea: Since grasping is a well-defined task in physics simulation - why can’t we just scale synthetic data generation and train a GenAI model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. Introducing… 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, antipodal pinch, and underactuated pinch grippers ✅ Generalization to both partial and complete 3D point clouds ✅ Generalization to both single-objects and cluttered scenes  ✅ Modular design relies on other robotics packages and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct any mistakes of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et. al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse We're also releasing: 🔹 Simulation-based grasp data generation workflows 🔹 Standardized formats and gripper definitions 🔹 Full training infrastructure 📄 arXiv: https://lnkd.in/gaYmcfz4 🌐 Website: https://lnkd.in/gGiKRCMX 💻 Code: https://lnkd.in/gYR77bEh A huge thank you to everyone involved in this journey — excited to hear the feedback from the community! Joint work with Clemens Eppner, Balakumar Sundaralingam, Yu-Wei Chao, Mark T. Carlson, Jun Yamada and other collaborators. Many thanks to Yichao Pan, Shri Sundaram, Spencer Huang, Buck Babich, Amit Goel for product management and feedback. #robotics #grasping #physicalAI #simtoreal

The new Newton physics engine is a game-changer for simulation. It’s not just faster — it’s truer to the real world. ✅ Why it matters: • Real robots operate in messy environments — dust, soil, cables, cloth, fluids. • Traditional simulators simplify or ignore those details. • Newton introduces physics intrinsics — friction, deformation, and material interaction — that make simulations behave like real life. ✅ What this means: • More accurate training data for AI and robotics. • More reliable behavior transfer from sim-to-real. • Fewer surprises when your model meets the real world. ✅ Key advantage: • GPU-native architecture runs thousands of environments at once. • Each environment obeys real-world physical laws. • Simulations now capture how dust swirls, fabric folds, or soil compresses — all in real time. ✅ The result: Simulation that doesn’t just look real — it behaves real. That’s the next frontier for robotics, AI, and digital twins. At FS Studio, we’re already integrating these new physics intrinsics into our synthetic data and digital twin workflows — pushing simulations to mirror the real world more faithfully than ever before. Check out Newton for yourself: https://lnkd.in/gaPVyGin

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

University of Michigan Researchers Introduce OceanSim: A High-Performance GPU-Accelerated Underwater Simulator for Advanced Marine Robotics Researchers from the University of Michigan have proposed OceanSim, a high-performance underwater simulator accelerated by NVIDIA parallel computing technology. Built upon NVIDIA Isaac Sim, OceanSim leverages high-fidelity, physics-based rendering, and GPU-accelerated real-time ray tracing to create realistic underwater environments. It bridges underwater simulation with the rapidly expanding NVIDIA Omniverse ecosystem, enabling the application of multiple existing sim-ready assets and robot learning approaches within underwater robotics research. Moreover, OceanSim allows the user to operate the robot, visualize sensor data, and record data simultaneously during GPU-accelerated simulated data generation. OceanSim utilizes NVIDIA’s powerful ecosystem, providing real-time GPU-accelerated ray tracing while allowing users to customize underwater environments and robotic sensor configurations. OceanSim implements specialized underwater sensor models to complement Isaac Sim’s built-in capabilities. These include an image formation model capturing water column effects across various water types, a GPU-based sonar model with realistic noise simulation for faster rendering, and a Doppler Velocity Log (DVL) model that simulates range-dependent adaptive frequency and dropout behaviors. For imaging sonar, OceanSim utilizes Omniverse Replicator for rapid synthetic data generation, establishing a virtual rendering viewport that retrieves scene geometry information through GPU-accelerated ray tracing..... Read full article: https://lnkd.in/gjTAkB2b Paper: https://lnkd.in/gEhq-SNQ