Accelerating Apache Spark^™ 3

Leveraging NVIDIA GPUs to Power the Next Era of Analytics and AI

GPU-accelerated libraries, DataFrame and APIs:

Layered on top of NVIDIA CUDA, RAPIDS is a suite of open-source software libraries and APIs that provide GPU parallelism and high-bandwidth memory speed through DataFrame and graph operations, achieving speedup factors of 50x or more on typical end-to-end data science workflows. For Spark 3.0, new RAPIDS APIs are used by Spark SQL and DataFrames for GPU accelerated memory efficient columnar data processing and query plans.
With Spark 3.0 the Catalyst query optimizer has been modified to identify operators within a query plan that can be accelerated with the RAPIDS API, and to schedule those operators on GPUs within the Spark cluster, when executing the query plan.
A new Spark shuffle implementation, built upon GPU accelerated communication libraries including Remote direct memory access (RDMA), dramatically reduces the data transfer among Spark processes. RDMA allows GPUs to communicate directly with each other, across nodes, at up to 100Gb/s, operating as if on one massive server.

GPU-aware Scheduling in Spark

Spark 3.0 adds integration with the cluster managers (YARN, Kubernetes, and Standalone) to request GPUs, and plugin points to allow it to be extended to run operations on the GPU. This makes GPUs easier to request and use for Spark application developers, allows for closer integration with deep learning and AI frameworks such as Horovod and TensorFlow on Spark, and allows for better utilization of GPUs.

Apache Spark^™ is a powerful execution engine for large-scale parallel data processing across a cluster of machines, which enables rapid application development and high performance.

In this ebook, learn how Spark 3 innovations make it possible to use the massively parallel architecture of GPUs to further accelerate Spark data processing.

Fill out the form below to download the ebook and learn about the following:

The data processing evolution, from Hadoop to GPUs and the NVIDIA RAPIDS™ library
Spark, what it is, what it does, and why it matters
GPU-acceleration in Spark
DataFrames and Spark SQL
A Spark regression example with a random forest classifier
An example of an end-to-end machine learning workflow GPU-accelerated with XGBoost

Download Now

Section

Section

First Name

Last Name

Business Email Address

Organization/University Name

Industry

Job Title

Location

Preferred Language

enterpriseOptIns hidden field

developerOptIns hidden field

Send me the latest news, announcements, and more from NVIDIA about Enterprise Business Solutions.

Send me the latest news, announcements, and more from NVIDIA about Developer Technology & Tools.

Send me the latest news, announcements, and more from NVIDIA about Enterprise Business Solutions and Developer Technology & Tools.

Send me the latest news, announcements, and more from NVIDIA about:

Enterprise Business Solutions

Developer Technology & Tools

(Optional). You can unsubscribe at any time.

NVIDIA Privacy Policy

I agree to the collection and processing of the above information by NVIDIA <span class="corporation-txt hidden">Corporation</span> for the purposes of research and event organization, and I have read and agree to <a href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/?deeplink=visiting-our-website" target="_blank">NVIDIA Privacy Policy</a>.

I agree that the above information will be transferred to NVIDIA Corporation in the United States and stored in a manner consistent with <a href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/?deeplink=visiting-our-website" target="_blank">NVIDIA Privacy Policy</a> due to necessities for research, event organization and corresponding NVIDIA internal management and system operation need. You may contact us by sending an email to <a href="mailto:privacy@nvidia.com">privacy@nvidia.com</a> to resolve related problems.

Accelerating Apache Spark™ 3

Download Now

Accelerating Apache Spark^™ 3