NVIDIA Digital Human Technologies Bring AI Game Characters To Life

At GDC 2024 and GTC 2024 we announced that leading AI developers across a wide range of industries are using NVIDIA digital human technologies to create lifelike avatars for commercial applications and dynamic game characters.

NVIDIA’s digital human suite of technologies includes NVIDIA Avatar Cloud Engine (ACE) for speech and animation, NVIDIA NeMo for language, and NVIDIA RTX for ray-traced rendering, enabling developers to build digital humans capable of AI-powered natural language interactions, making conversations more realistic and dynamic. 

Users experience NVIDIA ACE technologies through developer creations made possible by our digital human platform partners. At the shows, developers and platform partners unveiled new digital human technology demos leveraging several NVIDIA AI technologies and NVIDIA ACE microservices. Keep reading to learn how Hippocratic AI, Inworld AI, Perfect World Games, RealityArts Studio and UneeQ are leveraging NVIDIA ACE, NVIDIA NeMo, and NVIDIA RTX technologies in their games, platforms, and applications.

Inworld AI and NVIDIA Collaborate On New Covert Protocol Tech Demo 

At GDC 2024, Inworld and NVIDIA collaborated on a new technology demo called Covert Protocol to showcase NVIDIA ACE technologies and the Inworld Engine. In the Covert Protocol tech demo, you act as a private detective who completes objectives based on the outcome of conversations with characters in the scene. Covert Protocol unlocks social simulation game mechanics with AI Digital Humans acting as custodians of crucial information, presenting challenges, and catalyzing key narrative developments. Digital humans play a critical role in the experience.

Powered by the Inworld Engine and leveraging NVIDIA ACE, each player's journey through Covert Protocol is unique. Players’ real-time decisions and strategic planning lead to different game outcomes, ensuring that no two playthroughs are alike. This level of AI-driven interactivity and player agency opens up new possibilities for emergent gameplay, where players must think on their feet and adapt their strategies in real-time to navigate the intricacies of the game world.

Built on Unreal Engine 5, Covert Protocol leverages Inworld’s SDK to make integrating with MetaHuman fast and simple, and incorporates NVIDIA ACE technologies, including NVIDIA Riva Automatic Speech Recognition (ASR) and NVIDIA Audio2Face (A2F) to augment Inworld’s speech and animation pipelines.

“The combination of NVIDIA ACE microservices and the Inworld Engine enables developers to create digital characters that can drive dynamic narratives, opening new possibilities for how gamers can decipher, deduce and play.” said Kylan Gibbs, CEO of Inworld AI.

To accelerate developer adoption, Inworld will be releasing Covert Protocol’s source code in the near future, enabling developers to learn from the tech demo, and use it to create their own innovations.

Ubisoft’s NEO NPCs Explores the Possibilities of Digital Humans in Games with Inworld AI & NVIDIA ACE Technologies 

Ubisoft is exploring new types of interactive gameplay with dynamic Non Player Characters (NPCs), a field that is being revolutionized with the emergence of generative AI technologies. “NEO NPCs” the product of their latest research and development project are advanced NPCs designed to interact in real time with players, their environment and other characters, opening up new possibilities for dynamic and emergent storytelling.

Ubisoft showcased the capabilities of these NEO NPCs through three separate demo scenarios. Each scenario focuses on different aspects of NPC behaviors, environmental and contextual awareness; a spectrum of real-time reactions and animations; as well as conversation memory, collaboration and strategic decision-making. Combined, these discoveries through the experiment pushes the boundaries of game design and immersion.

With NEO NPCs, Ubisoft’s narrative and animation designers blended the reliability of scripted game elements with the dynamism of spontaneous player interactions, nudging the future of game development towards new horizons.

Leveraging Inworld AI technology, Ubisoft’s narrative team created two NEO NPCs, Bloom and Iron, who have their own background story, knowledge base, and different conversational styles to establish these unique characters in the demo universe.   

Inworld technology also provides the NEO NPCs with intrinsic knowledge of their surroundings, as well as interactive responses courtesy of Inworld’s LLM. NVIDIA Audio2Face provides facial animations and lip synching for the two NPCs’ faces in real-time.

World of Jade Dynasty & Unawake Also Incorporating Audio2Face Into Development Pipeline 

Perfect World Games’ World of Jade Dynasty is an upcoming Unreal Engine 5 massively multiplayer online role-playing game, and sequel to the firm’s incredibly popular Jade Dynasty.

A new video captured in-game demonstrates offline-rendered NVIDIA Audio2Face technology that generates accurate lip syncing in both Chinese and English, allowing script writers to easily add new voiced content to World of Jade Dynasty. Audio2Face can be used to easily support multi-language versions of the game.

In RealityArts Studio and Toplitz Productions’ Unreal Engine 5-powered Unawake, you decide the fate of humanity amidst the turmoil of merciless wars between nightmarish creatures from Heaven and Hell. Deliver punishment or absolution to your opponents in this action-adventure as you fight them in intense first-person melee skirmishes. You are the last bastion.

Unawake leveraged Audio2Face to create the characters’ facial animations during gameplay and in high-fidelity cinematics. Watch the new Unawake video below to see Audio2Face in action, and stay tuned for the game’s release later this year also featuring DLSS 3 with Frame Generation.

NVIDIA ACE Technologies: Building Blocks For Digital Humans and Virtual Assistants With Hippocratic AI & UneeQ

To showcase the new capabilities of NVIDIA digital human technologies, we’ve worked with leading companies, including Hippocratic AI and UneeQ, on a series of new technology demonstrations.

HippocraticAI has created a safety-focused, LLM-powered, task-specific Healthcare Agent. The agent calls patients on the phone, follows up on care coordination tasks, delivers preoperative instructions, performs post-discharge management and much more. For GTC, NVIDIA collaborated with HippocraticAI to extend its solution to use NVIDIA ACE microservices, NVIDIA Audio2Face along with NVIDIA Animation graph and NVIDIA Omniverse Streamer Client to show the potential of a generative AI healthcare agent avatar.

“Our digital assistants provide helpful, timely and accurate information to patients worldwide,” said Munjal Shah, cofounder and CEO of Hippocratic AI. “NVIDIA ACE technologies bring them to life with cutting-edge visuals and realistic animations that help better connect to patients.”

UneeQ is an autonomous digital human platform specialized in creating AI-powered avatars for customer service and interactive applications. UneeQ’s digital humans represent brands online, communicating with customers in real time to give them confidence in their purchases. UneeQ integrated NVIDIA Audio2Face microservice into its platform and combined it with Synanim ML to create highly realistic avatars for a better customer experience and engagement. 

“UneeQ combines NVIDIA animation AI with our own Synanim ML synthetic animation technology to deliver real-time digital human interactions that are emotionally responsive and deliver dynamic experiences powered by conversational AI,” said Danny Tomsett, CEO and Founder at UneeQ.

NVIDIA is at the forefront of Generative AI, and developers and platforms embracing NVIDIA ACE include: Convai, Cyber Agent, Data Monsters, Deloitte, HippocraticAI, IGOODI, Inworld AI, Media.Monks, miHoYo, NetEase Games, Perfect World, Openstream, OurPalm, Quantiphi, Rakuten Securities, Slalom, SoftServe, Tencent, Top Health Tech, Ubisoft, UneeQ and Unions Avatars. Head to their websites to discover how they’re leveraging digital human technology to create new possibilities and enhance customer experiences.

NVIDIA ACE: Bring Digital Characters To Life 

Top game and digital human developers are pioneering ways ACE and generative AI technologies can be used to transform interactions between players and NPCs in games and applications. Learn more about NVIDIA ACE, future developments, and early access programs. And discover more uses of NVIDIA RTX technology and GeForce RTX GPUs in the latest games by heading to GeForce.com.