Synthetic Data Generation (SDG)

Physical AI models allow autonomous systems to perceive, understand, interact with, and navigate the physical world. Synthetic data is critical for training and testing physical AI models.

World foundation models (WFMs) utilize diverse input data, including text, images, videos, and movement information, to generate and simulate virtual worlds with remarkable accuracy.   

WFMs are characterized by their exceptional generalization capabilities, requiring minimal fine-tuning for various applications. They serve as the cognitive engines for robots and autonomous vehicles, leveraging their comprehensive understanding of real-world dynamics. To achieve this level of sophistication, WFMs rely on vast amounts of training data. 

WFM development benefits significantly from generating infinite synthetic data through physically accurate simulations. This approach not only accelerates the model training process but also enhances the models' ability to generalize across diverse scenarios. Domain randomization techniques further augment this process by allowing for the manipulation of numerous parameters such as lighting, background, color, location, and environment—variations that would be nearly impossible to capture comprehensively from real-world data alone. 

Robot learning is a collection of algorithms and methodologies that help a robot learn new skills, such as manipulation, locomotion, and classification, in either a simulated or real-world environment. Reinforcement learning, imitation learning, and diffusion policy are the key methodologies applied to train robots.  

One important skill for robots is manipulation—picking things up, sorting them, and putting them together—like you see in factories. Real-world human demonstrations are typically used as inputs for training. However, collecting a large and diverse set of data is quite expensive. With a handful of human demonstrations, developers can generate synthetic motions in simulated environments, speeding up the robot training process.

To achieve this, users can first employ GR00T-Teleop to collect a small set of human demonstrations using Apple Vision Pro (AVP). The recorded demonstrations are then used to generate a large set of synthetic motions using GR00T-Mimic. Next, they use GR00T-Gen, built on NVIDIA Omniverse™ and NVIDIA Cosmos™, for domain randomization and 3D-to-real augmentation to generate an exponentially large and diverse set of training data for imitation learning. 

Software-in-loop (SIL) is a crucial testing stage for AI-powered robots and autonomous vehicles, where the control software is tested in a simulated environment instead of on real hardware.

Synthetic data generated from simulation ensures accurate modeling of real-world physics, including sensor inputs, actuator dynamics, and environmental interactions. This also provides a way to capture rare scenarios that are dangerous to collect in the real world. This ensures that the robot software stack in simulation behaves as it would on the physical robot, allowing for thorough testing and validation without the need for physical hardware.  

Mega is an Omniverse Blueprint for developing, testing, and optimizing physical AI and robot fleets at scale in a digital twin before deployment into real-world facilities.

These simulated robots can carry out tasks by perceiving and reasoning in environments. They are capable of planning next motions and then taking actions that are simulated in the digital twin. Synthetic data from these simulations is fed back into the robot brains. The robot brains perceive the results deciding the next action, and this cycle continues with Mega precisely tracking the state and position of all the assets in the digital twin.

Generative models can be used to bootstrap and augment synthetic data-generation processes. Text-to-3D models enable the creation of 3D assets for populating a 3D simulation scene. Text-to-image generative AI models can also be used to modify and augment existing images, either generated from simulations or collected in the real world through procedural inpainting or outpainting.

Text-to-text generative AI models, such as Evian 2 405B and Nemotron-4 340B, can be used to generate synthetic data to build powerful LLMs for healthcare, finance, cybersecurity, retail, and telecom. 

Evian 2 405B and Nemotron-4 340B provide an open license, giving developers the rights to own and use the generated data in their academic and commercial applications.