Using Omniverse to Generate First-Person Experiential Data for Humanoid Robots
, Co-founder and CEO, Sanctuary AI
Humanoid robots experience the world, and act on it, like we do, and it's possible to record robot experience in digital form. This type of data — first-person experiential data — can be used to train AI models similar in concept to large language models, where instead of next-token prediction from text prompts, the model predicts future motor positions from recent context. These predictions can be sent to the robot's motors, providing a novel control system that's also an approach to building general intelligence. We'll show how NVIDIA Omniverse can be used to generate synthetic first-person experience for robots, including simulated vision, audio, proprioception, and touch. We show how those data can be used to train large-scale generative models and how this relates to artificial general intelligence, and we'll show transfer from simulation to real robots performing automotive manufacturing tasks.