Physical AI enables autonomous machines like robots and self-driving cars to perceive, understand, and perform complex actions in the real (physical) world. It is also often referred to as “generative physical AI” due to its ability to generate insights and actions to execute on.
Generative AI models—large language models such as GPT and Llama—are trained on enormous amounts of text and image data, largely gathered from the Internet. These AIs have astonishing capabilities in producing human language and abstract concepts, but they're limited in their grasp of the physical world and its rules.
Generative physical AI extends current generative AI with understanding of spatial relationships and physical behavior of the 3D world we all live in. This is done by providing additional data that contains information about the spatial relationships and physical rules of the real world during the AI training process.
The 3D training data is generated from highly accurate computer simulations, which serve as both a data source and an AI training ground.
Physically-based data generation starts with a digital twin of a space, such as a factory. In this virtual space, sensors and autonomous machines like robots are added. Simulations that mimic real-world scenarios are performed, and the sensors capture various interactions like rigid body dynamics—such as movement and collisions—or how light interacts in an environment.
Reinforcement learning teaches autonomous machines skills in a simulated environment to perform in the real world. It allows autonomous machines to learn skills safely and quickly through thousands or even millions of acts of trial and error.
This learning technique rewards a physical AI model for successfully completing desired actions in the simulation so the model continuously adapts and improves. With repeated reinforcement learning, autonomous machines eventually adapt to new situations and unforeseen challenges appropriately, preparing them to operate in the real world. Over time, an autonomous machine can develop sophisticated fine motor skills needed for real-world applications, such as neatly packing boxes, helping to build vehicles, or navigating environments unassisted.
Previously, autonomous machines were unable to perceive and sense the world around them. But with generative physical AI, robots can be built and trained to seamlessly interact with and adapt to their surroundings in the real world.
To build physical AI, teams need powerful, physics-based simulations that provide a safe, controlled environment for training autonomous machines. This not only enhances the efficiency and accuracy of robots in performing complex tasks, but also facilitates more natural interactions between humans and machines, improving accessibility and functionality in real-world applications.
Generative physical AI is unlocking new capabilities that will transform every industry. For example:
Robots: With physical AI, robots demonstrate significant advancements in operational capabilities within various settings.
Autonomous Vehicles (AVs): AVs use sensors to perceive and understand their surroundings, enabling them to make informed decisions in various environments, from open freeways to urban cityscapes. By training AVs on physical AI, it allows the AVs to more accurately detect pedestrians, respond to traffic or weather conditions, and autonomously navigate lane changes, effectively adapting to a wide range of unexpected scenarios.
Smart Spaces: Physical AI is enhancing the functionality and safety of large indoor spaces like factories and warehouses, where daily activities involve a steady traffic of people, vehicles, and robots. Using fixed cameras and advanced computer vision models, teams can enhance dynamic route planning and optimize operational efficiency by tracking multiple entities and activities within these spaces. Additionally, they prioritize human safety by accurately perceiving and understanding complex, large-scale environments.
Building the next generation of autonomous machines using physical AI involves a coordinated process across multiple, specialized computers: