Accelerate Apache Spark 3™ data science pipelines—without code changes—and speed up data processing and model training while substantially lowering infrastructure costs.
Accelerate the performance of data preparation tasks to quickly move to the next stage of the pipeline. This allows models to be trained faster, while freeing up data scientists and engineers to focus on the most critical activities.
Do more with less: Spark on NVIDIA® GPUs completes jobs faster with less hardware when compared to CPUs, saving organizations time as well as on-premises capital costs or operational costs in the cloud.
Use NVIDIA AI Enterprise, an end-to-end AI software platform including RAPIDS Accelerator, to speed time to production by accelerating the end to end AI pipeline from data prep and processing to, model training, simulation, and inference at scale
Given the “embarrassingly parallel” nature of many data processing tasks, it’s only natural that GPU architecture should be used for Spark data processing queries, similar to how a GPU can accelerate deep learning workloads in AI. GPU acceleration is transparent to the developer and doesn’t require code changes to obtain benefits. Three key advancements in Spark 3 have contributed to delivering transparent GPU acceleration:
NVIDIA CUDA® is a revolutionary parallel computing platform that supports accelerating computational operations on NVIDIA GPU architecture. RAPIDS, incubated at NVIDIA, is a suite of open-source libraries layered on top of CUDA that enables GPU acceleration of data science pipelines.
NVIDIA has created a RAPIDS Accelerator for Spark 3 that intercepts and accelerates extract, transform and load pipelines by dramatically improving the performance of Spark SQL and DataFrame operations.
Spark 3 provides columnar processing support in the Catalyst query optimizer, which is what the RAPIDS Accelerator plugs into to accelerate SQL and DataFrame operators. When the query plan is executed, those operators can then be run on GPUs within the Spark cluster.
NVIDIA has also created a new Spark shuffle implementation that optimizes the data transfer between Spark processes. This shuffle implementation is built on GPU-accelerated communication libraries, including UCX, RDMA, and NCCL.
Spark 3 recognizes GPUs as a first-class resource along with CPU and system memory. This allows Spark 3 to place GPU-accelerated workloads directly onto servers containing the necessary GPU resources as they are needed to accelerate and complete a job.
NVIDIA engineers have contributed to this major Spark enhancement, enabling the launch of Spark applications on GPU resources in Spark standalone, YARN, and Kubernetes clusters.
RAPIDS Accelerator for Apache Spark is available with NVIDIA AI Enterprise. Get optimized performance for Spark deployments with full access to enterprise-grade support, security, and stability on certified platforms across on prem to cloud – including Amazon EMR, Google Cloud Dataproc and Databricks. Take advantage of guaranteed response times, priority security notifications, and access to data science experts from NVIDIA.
To unlock the value of AI-powered big data and learn more about the next evolution of Apache Spark, download NVIDIA’s new e-book, “Accelerating Apache Spark 3.x – Leveraging NVIDIA GP."Us to Power the Next Era of Analytics and AI.
