Robotics Simulation

NVIDIA Isaac Sim™ is a reference application, built on NVIDIA Omniverse™, that lets you build, train, test, and validate AI-powered robots such as humanoids, autonomous mobile robots (AMRs), and robot arms—entirely in simulated environments.

1. Importing Assets: Use existing assets that have been built in 3D CAD or DCC software tools. These assets need to be converted to Universal Scene Description (OpenUSD) prior to use in Isaac Sim.
2. Creating Environments: Once the relevant assets are brought in, create a virtual environment, such as a warehouse or a factory. The goal here is to represent the space as accurately as possible to its real world, including the colors, textures, and lighting.
3. Adding Robots: Once the scene has been set up, robot models can be brought in through Universal Robot Description Format, or URDF. URDF also includes visual meshes and the prim hierarchies (child-parent relationships) and information collision meshes, joints, and sensors.
4. Adding Physics and Sensors: To interact properly in the real world, physical attributes need to be assigned to the robots. Physics simulations for Rigid Body and Deformable Bodies, along with Articulations made possible by the NVIDIA® PhysX® physics engine, allow robots to master the kinematics of their environment. Both visual (e.g. camera) and non-visual (lidar, radar, IMU, etc.) will also need to be added to capture the robot’s behavior.
5. Interaction: The final step in this process is to simulate the robot or robots in the various spatio-temporal scenarios.

Simulation can unlock novel use cases by bootstrapping the training of foundational models or speed up the fine-tuning process of pretrained AI models with synthetic data generation, or SDG. It can consist of text, 2D, or 3D images in the visual and non-visual spectrum, and even motion data that can be used in conjunction with real-world data to train multimodal physical AI models.

Domain randomization is a key step in the SDG workflow, where many parameters in a scene can be changed to generate a diverse dataset—from location, to color, to textures to lighting of the objects. Augmentation in the post-processing phase further diversifies generated data by adding defects such as localized blurring, pixelation, randomized cropping, skewing, and blending.

Additionally, the images generated are automatically annotated and can include RGB, bounding boxes, instance and semantic segmentation, depth, depth point cloud, lidar point cloud, and more.

Robot learning is critical to ensure that it can perform robust skills repeatedly and efficiently in the physical world. High-fidelity simulation provides a virtual training ground for robots to hone their skills through trial and error or through imitation. This ensures that the robot's learned behaviors in simulation are more easily transferable to the real world.

NVIDIA Isaac™ Lab, an open-source, unified and modular framework for robot learning built on NVIDIA Isaac Sim, simplifies common workflows in robotics learning such as reinforcement learning, learning from demonstrations, and motion planning.

Software-in-loop (SIL) is a critical testing and validation stage in the development of software for physical AI-powered robotics systems. In SIL, the software that controls the robot is tested in a simulated environment rather than on the actual hardware.

SIL with simulation ensures accurate modeling the physics of the real world, including sensor inputs, actuator dynamics, and environmental interactions. This ensures that the robot software stack behaves in the simulation as it would on the physical robot, improving the validity of the testing results.

Synthetic data generation, robot learning, and robot testing are highly interdependent workflows and require careful orchestration across a heterogeneous infrastructure. Robotic workflows also require developer-friendly specification that removes the complexity with infrastructure setup, easy ways to trace data and model lineage, and a secure way to deploy these workloads.

NVIDIA OSMO gives you a cloud-native orchestration platform for scaling complex, multi-stage, and multi-container ‌robotics workloads, across on-premises, private, and public clouds.