31:00
NVIDIA NIM Microservices
Designed for rapid, reliable deployment of accelerated generative AI inference anywhere.
Overview
What Is NVIDIA NIM?
NVIDIA NIM™ provides prebuilt, optimized inference microservices for rapidly deploying the latest AI models on any NVIDIA-accelerated infrastructure—cloud, data center, workstation, and edge.
Accelerate AI Deployment With NVIDIA NIM
NVIDIA NIM combines the ease of use and operational simplicity of managed APIs with the flexibility and security of self-hosting models on your preferred infrastructure. NIM microservices come with everything AI teams need—the latest AI foundation models, optimized inference engines, industry-standard APIs, and runtime dependencies—prepackaged in enterprise-grade software containers ready to deploy and scale anywhere.
Benefits
Enterprise Generative AI That Does More for Less
Easy, enterprise-grade microservices built for high-performance AI—designed to work seamlessly and scale affordably. Experience the fastest time to value for AI agents and other enterprise generative AI applications powered by the latest AI models for reasoning, simulation, speech, and more.
Ease of Use
Accelerate innovation and time to market with prebuilt, optimized microservices for the latest AI models. With standard APIs, models can be deployed in five minutes and easily integrated into applications.
Enterprise Grade
Deploy enterprise-grade microservices that are continuously managed by NVIDIA through rigorous validation processes and dedicated feature branches—all backed by NVIDIA enterprise support, which also offers direct access to NVIDIA AI experts.
Performance and Scale
Improve TCO with low-latency, high-throughput AI inference that scales with the cloud, and achieve the best accuracy with support for fine-tuned models out of the box.
Portability
Deploy anywhere with prebuilt, cloud-native microservices ready to run on any NVIDIA-accelerated infrastructure—cloud, data center, and workstation—and scale seamlessly on Kubernetes and cloud service provider environments.
Demo
Build AI Agents With NIM
Learn how to set up two AI agents—one for content generation and another for digital graphic design—and see how easy it is to get up and running with NIM microservices.
Technology
Building Blocks for Agentic AI
Benchmarks
Boost Throughput With NIM
NVIDIA NIM provides optimized throughput and latency out of the box to maximize token generation, support concurrent users at peak times, and improve responsiveness. NIM microservices are continuously updated with the latest optimized inference engines, boosting performance on the same infrastructure over time.
Configuration: Llama 3.1 8B instruct, 1x H100 SXM; concurrent requests: 200. NIM ON: FP8, throughput 1201 tokens/s, ITL 32ms. NIM OFF: FP8, throughput 613 tokens/sec, ITL 37ms.
Models
Build With the Leading Open Models
Get optimized inference performance for the latest AI models to power multimodal agentic AI with reasoning, language, retrieval, speech, image, and more. NIM comes with accelerated inference engines from NVIDIA and the community, including NVIDIA® TensorRT™, TensorRT-LLM, and more—prebuilt and optimized for low-latency, high-throughput inferencing on NVIDIA-accelerated infrastructure.
Features
The Easy Button for AI Development and Deployment
Designed to run anywhere, NIM inference microservices expose industry-standard APIs for easy integration with enterprise systems and applications and scale seamlessly on Kubernetes to deliver high-throughput, low-latency inference at cloud scale.
Deploy NIM
Deploy NIM for your model with a single command. You can also easily run NIM with fine-tuned models.
Run Inference
Get NIM up and running with the optimal runtime engine based on your NVIDIA-accelerated infrastructure.
Build
Integrate self-hosted NIM endpoints with just a few lines of code.
Deploy
Run
Build
docker run nvcr.io/nim/publisher_name/model_name
curl -X 'POST' \
'http://0.0.0.0:8000/v1/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"model" : "model_name",
"prompt" : "Once upon a time",
"max_tokens" : 64
}'
import openai
client = openai.OpenAI(
base_url = "YOUR_LOCAL_ENDPOINT_URL",
api_key="YOUR_LOCAL_API_KEY"
)
chat_completion = client.chat.completions.create(
model="model_name",
messages=[{"role" : "user" , "content" : "Write me a love song" }],
temperature=0.7
)
Use Cases
How NIM Is Being Used
See how NVIDIA NIM supports industry use cases, and jump-start your AI development with curated examples.
-
AI Virtual Assistants
-
Document Intelligence
-
Hyperpersonalized Shopping
-
3D Product Configurators
AI Virtual Assistants
Enhance customer experiences and improve business processes with generative AI.
Starting Options
Ways to Get Started With NVIDIA NIM
Resources
The Latest NVIDIA NIM Resources
-
Blogs
-
Sessions
-
Courses
-
Videos