Sony Pictures Animation

Bringing 2D and 3D Worlds Together With NVIDIA Omniverse Enterprise

Objective

Sony Pictures Animation has made great strides in improving their pre-production workflow, allowing their artists to have greater control and creativity when transitioning their 2D storyboards into a 3D space. They developed an application called FlixiVerse on NVIDIA Omniverse™, and leveraged the platform's existing features such as OpenUSD and Nucleus’ auto-updating capabilities. This has allowed the artists to easily move their 3D models to Nucleus in minutes, and they can navigate through 3D scenes using a game controller to manipulate the camera.

Customer

Sony Pictures Animation

Use Case

Media and Entertainment

Technology

NVIDIA Omniverse Enterprise
NVIDIA RTX™
FlixiVerse
Universal Scene Description (OpenUSD)

Sony Pictures Animation leveraged NVIDIA Omniverse to develop an internal application called FlixiVerse, which allows non-3D-fluent artists and directors to easily navigate a 3D environment, making better-informed creative decisions and accelerating the pre-production workflow.

Significant optimization for the pre-production workflow.

Time for 4X more shot iterations compared to previous films.

FlixiVerse syncs files automatically in minutes versus days compared to previous workflow.
Using the Omniverse extension library saved weeks in development time.

Image Credit: Courtesy of Sony Pictures Animation

The studio developed FlixiVerse, an app built on Omniverse Enterprise and powered by NVIDIA RTX, allowing 2D artists to accelerate creative workflows with more iterations and easier transitions into 3D environments.

The studio focuses on the front-end of animation pre-production, partnering with sister studio Sony Pictures Imageworks for feature shot production. This allows their team to focus on style and innovation in storyboarding, visual development, and editorial while closely collaborating on the final production. On average a 90-minute film will have around 120,000 storyboards, with over 2000 updates to the script and 25,000 unique designs.

To provide downstream artists with as much creative freedom as possible, Sony Pictures Animation started to explore new ways to optimize their pre-production workflow.

“When artists are working on storyboard panels, they’re mostly working on 2D drawings. It looks beautiful and works great as a storyboard, but transitioning these 2D boards into a 3D space can be challenging. Often a shot doesn’t work in 3D, or a camera angle cannot be achieved. Additionally, 2D panels can be limiting for creative expression in scenes such as action shots,” said Yiotis Katsambas, the executive director of technology at Sony Pictures Animation.

"It felt like we needed to give artists more control of moving to the 3D environment, We didn't want to limit the artists—we wanted to put technology in place that allowed them to use the tools and apps that make sense for them, while still giving them as much flexibility as possible."

Image Credit: Courtesy of Sony Pictures Animation

A Portal into the Third Dimension

The goal was to immerse non-3D-fluent artists and directors into a simplified 3D view so they can make better-informed creative decisions, and iterate on layout in a 3D space. Achieving this bridge would also allow production teams to capture data from 3D pre-production, such as lenses and camera information, to help the transition into production.

The team tried a few ways to address this, including building a desktop app. Even then, they faced challenges such as models taking days to update, artists not being familiar with the 3D tools, or not having powerful enough machines to run the desktop applications.

They then turned their attention to building on Omniverse after seeing a demo at NVIDIA GTC. A small team led by engineer Nikolas Ladas began developing an internal application called FlixiVerse—this app would be their "portal to the third dimension."

FlixiVerse allows the storyboarding team to easily navigate a 3D environment using a simple viewer, so anyone can enter a scene and iterate in 3D. FlixiVerse syncs files automatically in minutes versus days compared to the previous workflow. It’s also web-based, leveraging a pool of PCs at the studio, along with a server powered by NVIDIA RTX 6000 GPUs. This makes it easy for artists to work and review in a browser without needing powerful machines.

Accelerating App Development with Existing Components and Extensions

When getting started building a kit-based application with Omniverse, the team quickly realized a few advantages—mainly through Omniverse’s modular extensibility and using existing components available as building blocks.

“Building Flixiverse using Omniverse Kit was a blast. It gives you a ready-to-use, real-time 3D renderer that can ray-trace, so a lot of what we had to implement was there from the get-go. Extending the base project was easy since all development is in Python.” said Nikolas Ladas, software engineer at Sony Pictures Animation.

Much of what FlixiVerse does, comes from kit plugins. This includes UI, working with USD data, communicating with Nucleus for storage, and streaming the application over the network using webRTC. Adding a plugin to Flixiverse was as simple as adding a line to the project's configuration file. For code that SPA had to write themselves, they looked to the documentation and the various example projects.

Omniverse’s OpenUSD (Universal Scene Description) architecture jump started development and interoperability with Sony’s existing pipeline. The team used extensions from the Omniverse extension library, such as Scene Optimizer, which allowed them to easily add functionality to convert complex 3D scenes into lightweight representations, saving‌ weeks in development time. Nucleus adds file transfer and auto-updates functionality, pulling models from Autodesk Maya.

"We created a script that would use Omniverse commands to take what's live in Maya, and export it to Omniverse Nucleus," said Katsambas. "Previously, it would take days to prepare and export files. Now it only takes minutes."

To simplify the control of the camera, the team integrated a PlayStation controller so they could easily move the camera around the scene. There's also a customized interface that allows their artists to tweak lights, shading, camera angles, and more. Now they can pull this information into production and help accelerate and optimize shot building.

Director Joaquim Dos Santos is a filmmaker with a background in storyboarding who used FlixiVerse during pre-production for the studio’s latest film, Spider-Man: Across the Spider-Verse. “FlixiVerse was invaluable to Joaquim’s process, now he can move around in 3D using a game controller, take snapshots, move models, and speed up his workflow. Additionally, production designers can see the pre-production design in a 3D world to get inspiration and design ideas,” Katsambas continued.

The team used a combination of an internal tool called Flixi, and their Omniverse app FlixiVerse in pre-production for Spider-Man: Across the Spider-Verse. At the end of pre-production, their final count of storyboard panels for the film was over 450,000—demonstrating almost 4X more iterations than the average count for previous films.

For next steps, Katsambas explains they plan to continue developing FlixiVerse, and that this is just the beginning. "We're just scratching the surface," he said. "We like the idea of Omniverse Nucleus, and there are plenty of opportunities to explore how we can creatively use it, such as connecting new applications for mo-cap to accelerate character staging, and look-dev tools."

Start developing interoperable 3D workflows and OpenUSD applications.

Get in Touch