Meet the challenges head on with NVIDIA® Tesla® GPUs and NVIDIA TensorRT™ platform, the world’s fastest, most efficient deep learning inference platform. NVIDIA’s inference platform supports all deep learning workloads and provides the optimal inference solution—combining the highest throughput, best efficiency, and best flexibility to power AI-driven experiences.
Watch to see how Microsoft is using it.
The Tesla V100 has 125 teraflops of inference performance per GPU. A single server with eight Tesla V100s can produce a petaflop of compute.
The Tesla P4 accelerates any scale-out server, offering an incredible 60X higher energy efficiency compared to CPUs.
The Tesla P40 offers great inference performance, INT8 precision, and 24GB of onboard memory for an amazing user experience.
NVIDIA TensorRT is a high-performance deep learning inference platform that can speed up applications such as recommenders, speech recognition, and machine translation by 40X compared to CPU-only architectures.
O NVIDIA Triton Inference Server, anteriormente conhecido como TensorRT Inference Server, é um software de código aberto que simplifica a implantação de modelos de deep learning na produção. O Triton Inference Server permite que as equipes implantem modelos treinados de AI em qualquer estrutura (TensorFlow, PyTorch, TensorRT Plan, Caffe, MXNet ou personalizada) a partir do armazenamento local, do Google Cloud Platform ou do AWS S3 em qualquer infraestrutura baseada em GPU ou CPU. Ele executa vários modelos simultaneamente em uma única GPU para maximizar a utilização e se integra ao Kubernetes para orquestração, métricas e dimensionamento automático.
Kubernetes on NVIDIA GPUs enables enterprises to scale up training and inference deployment to multi-cloud GPU clusters seamlessly. With Kubernetes, GPU-accelerated deep learning and high performance computing (HPC) applications can be deployed to multi-cloud GPU clusters instantly.
NVIDIA DeepStream is an application framework for the most complex Intelligent Video Analytics (IVA) applications. Developers can now focus on building core deep learning networks rather than designing end-to-end applications from scratch given its modular framework and hardware-accelerated building blocks.
NVIDIA Tesla T4 has the world’s highest inference efficiency, up to 40X more compared to CPUs. T4 can analyze up to 39 simultaneous HD video streams in real-time using dedicated hardware-accelerated video transcode engines. Bringing all this performance in just 70 watts (W) makes NVIDIA T4 the ideal inference solution for mainstream servers at the edge.
Tesla V100 GPUs powered by NVIDIA Volta™ give data centers a dramatic boost in throughput for deep learning workloads to extract intelligence from today’s tsunami of data. A server with a single Tesla V100 can replace up to 50 CPU-only servers for deep learning inference workloads, so you get dramatically higher throughput with lower acquisition cost.
NVIDIA TensorRT optimizer and runtime engines deliver high throughput at low latency for applications such as recommender systems, speech recognition and image classification. With TensorRT, models trained in 32-bit or 16-bit data can be optimized for INT8 operations on Tesla T4 and P4, or FP16 on Tesla V100. NVIDIA DeepStream SDK taps into the power of Tesla GPUs to simultaneously decode and analyze video streams.
NVIDIA Triton Inference Server delivers high throughput data center inference and helps you get the most from your GPUs. Delivered in a ready-to-run container, NVIDIA Triton Inference Server is a microservice that concurrently runs models from Caffe2, NVIDIA TensorRT, TensorFlow, and any framework that supports the ONNX standard on one or more GPUs.
| Tesla T4: The World's Most Advanced Inference Accelerator | Tesla V100: The Universal Data Center GPU | Tesla P4 for Ultra-Efficient, Scale-Out Servers | Tesla P40 for Inference-Throughput Servers | |
|---|---|---|---|---|
| Single-Precision Performance (FP32) | 8.1 TFLOPS | 14 TFLOPS (PCIe) 15.7 teraflops (SXM2) |
5.5 TFLOPS | 12 TFLOPS |
| Half-Precision Performance (FP16) | 65 TFLOPS | 112 TFLOPS (PCIe) 125 TFLOPS (SXM2) |
— | — |
| Integer Operations (INT8) | 130 TOPS | — | 22 TOPS* | 47 TOPS* |
| Integer Operations (INT4) | 260 TOPS | — | — | — |
| GPU Memory | 16GB | 32/16GB HBM2 | 8GB | 24GB |
| Memory Bandwidth | 320GB/s | 900GB/s | 192GB/s | 346GB/s |
| System Interface/Form Factor | Low-Profile PCI Express Form Factor | Dual-Slot, Full-Height PCI Express Form Factor SXM2 / NVLink | Low-Profile PCI Express Form Factor | Dual-Slot, Full-Height PCI Express Form Factor |
| Power | 70 W | 250 W (PCIe) 300 W (SXM2) |
50 W/75 W | 250 W |
| Hardware-Accelerated Video Engine | 1x Decode Engine, 2x Encode Engines | — | 1x Decode Engine, 2x Encode Engines | 1x Decode Engine, 2x Encode Engines |
Lower response time for speech recognition apps while maintaining accuracy on NVIDIA Tesla GPUs running TensorRT software.
Maximize throughput efficiency for image and video processing workloads with NVIDIA DeepStream SDK and Tesla GPUs.
Increase recommender prediction accuracy with deep learning based neural collaborative filtering apps running on NVIDIA GPU platforms.
The Tesla V100, T4, and P40 are available now for deep learning inference.
