31:00
NVIDIA NIM Microservices
Designed for rapid, reliable deployment of accelerated generative AI inference anywhere.
Overview
What Is NVIDIA NIM?
NVIDIA NIM™ provides prebuilt, optimized inference microservices that let you deploy the latest AI foundation models with security and stability on any NVIDIA-accelerated infrastructure— cloud, data center, and workstation.
Generative AI Deployment, Accelerated With NVIDIA NIM
NVIDIA NIM combines the ease of use and operational simplicity of managed APIs with the flexibility and security of self-hosting models on your preferred infrastructure. NIM microservices come with everything AI teams need—the latest AI foundation models, optimized inference engines, industry-standard APIs, and runtime dependencies—prepackaged in enterprise-grade software containers ready to deploy and scale anywhere.
Benefits
Generative AI for Enterprises That Does More for Less
Get the best of both worlds with easy, enterprise-grade microservices built for high-performance AI—designed to work seamlessly and scale affordably. Experience the fastest time to value for enterprise AI agents and other generative AI domains such as reasoning, simulation, speech, and more.
Ease of Use
Accelerate time to market with optimized, prebuilt, cloud-native microservices and empower enterprise developers with industry-standard APIs and tools tailored for enterprise needs.
Security and Manageability
Securely control generative AI applications and data with self-hosted deployment on your choice of infrastructure. Take advantage of enterprise-grade support, including dedicated feature branches, rigorous validation processes, and direct access to NVIDIA AI experts.
Performance and Scale
Improve TCO with low-latency, high-throughput AI inference that scales in the cloud, and achieve the best accuracy with support for fine-tuned models out of the box.
Portability
Deploy anywhere with prebuilt microservices ready to run on any NVIDIA-accelerated infrastructure—cloud, data center, and workstation—and scale seamlessly on Kubernetes and cloud service provider environments.
Benchmarks
Boost Throughput With NIM
NVIDIA NIM provides optimized throughput and latency out of the box to maximize token generation, support concurrent users at peak times, and improve responsiveness.
Configuration: Llama 3.1 8B instruct, 1x H100 SXM; input 20000 tokens, output 2000 tokens, concurrent requests: 200. NIM ON: FP8, throughput 1201 tokens/s, ITL 32ms. NIM OFF: FP8, throughput 613 tokens/sec, ITL 37ms.
Models
Build With the Leading Open Models
Get optimized inference performance for the latest AI foundation models. NIM comes with accelerated inference engines from NVIDIA and the community, including NVIDIA® TensorRT™, TensorRT-LLM, and more—prebuilt and optimized for low-latency, high-throughput inferencing on NVIDIA-accelerated infrastructure.
Features
Enterprise Generative AI Is Just an API Call Away
Designed to run anywhere, NIM inference microservices expose industry-standard APIs for easy integration with enterprise systems and applications and scale seamlessly on Kubernetes to deliver high-throughput, low-latency inference at cloud scale.
Deploy NIM
Deploy NIM for your model with a single command. You can also easily run NIM with fine-tuned models.
Run Inference
Get NIM up and running with the optimal runtime engine based on your NVIDIA-accelerated infrastructure.
Build
Integrate self-hosted NIM endpoints with just a few lines of code.
Deploy
Run
Build
docker run nvcr.io/nim/publisher_name/model_name
curl -X 'POST' \
'http://0.0.0.0:8000/v1/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"model" : "model_name",
"prompt" : "Once upon a time",
"max_tokens" : 64
}'
import openai
client = openai.OpenAI(
base_url = "YOUR_LOCAL_ENDPOINT_URL",
api_key="YOUR_LOCAL_API_KEY"
)
chat_completion = client.chat.completions.create(
model="model_name",
messages=[{"role" : "user" , "content" : "Write me a love song" }],
temperature=0.7
)
Use Cases
How NIM Is Being Used
See how NVIDIA NIM supports industry use cases, and jump-start your AI development with curated examples.
-
AI Virtual Assistants
-
Document Intelligence
-
Hyperpersonalized Shopping
-
3D Product Configurators
AI Virtual Assistants
Enhance customer experiences and improve business processes with generative AI.
Starting Options
Ways to Get Started With NVIDIA NIM
Resources
The Latest NVIDIA NIM Resources
-
Blogs
-
Sessions
-
Courses
-
Videos