Pandas

Pandas is the most popular software library for data manipulation and data analysis for the Python programming language.

As an open-source software library built on top of Python specifically for data manipulation and analysis, Pandas offers data structure and operations for  powerful, flexible, and easy-to-use data analysis and manipulation. Pandas strengthens Python by giving the popular programming language the capability to work with spreadsheet-like data enabling fast loading, aligning, manipulating, and merging, in addition to other key functions. Pandas is prized for providing highly optimized performance when back-end source code is written in C or Python.

The name ‘Pandas’ comes from the econometrics term ‘panel data’ describing data sets that include observations over multiple time periods. The Pandas library was created as a high-level tool or building block for doing very practical real-world analysis in Python. Going forward, its creators intend Pandas to evolve into the most powerful and most flexible open-source data analysis and data manipulation tool for any programming language.

What some have called a ‘game changer’ for analyzing data with Python, Pandas ranks among the most popular and widely used tools for so-called data wrangling, or munging. This describes a set of concepts and a methodology used when taking data from unusable or erroneous forms to the levels of structure and quality needed for modern analytics processing. Pandas excels in its ease of working with structured data formats such as tables, matrices, and time series data. It also works well with other Python scientific libraries.

Included in the Pandas open-source library are DataFrames, which are two-dimensional array-like data tables in which each column contains values of one variable and each row contains one set of values from each column. Data stored in a DataFrame can be of numeric, factor, or character types. Pandas DataFrames are also thought of as a dictionary or collection of series objects.

Data scientists and programmers familiar with the R programming language for statistical computing know that DataFrames are a way of storing data in grids that are easily overviewed. This means that Pandas is chiefly used for machine learning in the form of DataFrames.

Pandas allows for importing and exporting tabular data in various formats, such as CSV or JSON  files. 

Pandas also allows for various data manipulation operations and for data cleaning features, including selecting a subset, creating derived columns, sorting, joining,  filling, replacing, summary statistics, and plotting.

According to organizers of the Python Package Index—a repository of software for the Python programming language—Pandas is well suited for working with several kinds of data, including:

• Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
• Ordered and unordered (not necessarily fixed-frequency) time series data
• Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels

Any other form of observational/statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure.

Again according to the Python Package Index organizers, Pandas delivers several key benefits to data scientists and developers alike, including:

• Easy handling of missing data (represented as NaN) in both floating point and non-floating point data
• Size mutability: columns can be inserted and deleted from DataFrames and higher-dimensional objects
• Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let series, DataFrame, etc. automatically align the data in computations
• Powerful, flexible group-by functionality to perform split-apply-combine operations on data sets for both aggregating and transforming data
• Making it easy to convert ragged, differently indexed data in other Python and Numpy data structures into DataFrame objects
• Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
• Intuitive merging and joining of data sets
• Flexible reshaping and pivoting of data sets
• Hierarchical labeling of axes (possible to have multiple labels per tick)
• Robust I/O tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving/loading data from the ultrafast HDF5 format
• Time series-specific functionality: date range generation and frequency conversion, moving window statistics, date shifting, and lagging

Additional benefits derived from the Pandas library include data alignment and integrated handling of missing data; data set merging and joining; reshaping and pivoting of data sets; hierarchical axis indexing to work with high-dimensional data in a lower-dimensional data structure; and label-based slicing.

Given that Pandas is built on top of the Python programming language, a brief review of the Python programming language is in order.

A favorite with data scientists owing to its ease-of-use, Python has evolved from its earliest roots in 1991 to be one of the most popular programming languages for web applications, data analysis, and machine learning. 

Python’s ease-of-use means even beginners can produce programs with relatively little up-front time investment owing to Python’s highly readable syntax. This means developers and data scientists spend more time-solving business problems and less time wrestling with language complexities.

Python runs on every significant operating system in use today, as well as major libraries in addition to Pandas. API services also have Python links or so-called wrappers. This allows Python to interface with other services and libraries.

In addition to its ease of use, Python has become a favorite for data scientists and machine learning developers for another good reason. With the availability today of data-handling libraries like Pandas and Numpy, and with data visualization tools like Seaborn and Matplotlib, Python is lingua franca for machine learning and the data scientists and developers building machine learning systems.

Pandas addresses the many shortcomings that data scientists often encounter when using languages associated with scientific and business research environments. In data science, working with data is usually sub-divided into multiple stages, including the aforementioned munging and data cleaning; analysis and modeling of data; and organizing the analysis into a form agreeable for plotting or display in tabular form. For these and other mission-critical data science tasks, Pandas excels.

A CPU consists of a few cores, optimized for sequential serial processing, whereas a GPU has a massively parallel architecture consisting of thousands of smaller, more efficient cores designed for handling multiple tasks simultaneously. GPUs are capable of processing data much faster than configurations containing CPUs alone. They’re also popular for their extraordinarily low price per flop (performance) and are addressing the compute performance bottleneck today by speeding up multi-core servers for parallel processing. 

GPUs have been responsible for the advancement of deep learning in the past several years, while ETL and traditional machine learning workloads continued to be written in Python—often with single-threaded tools like Scikit-Learn or large, multi-CPU distributed solutions like Spark.

NVIDIA developed RAPIDS^™—an open-source data analytics and machine learning acceleration platform—for executing end-to-end data science training pipelines completely in GPUs. It relies on NVIDIA^® CUDA^® primitives for low-level compute optimization, but exposes that GPU parallelism and high memory bandwidth through user-friendly Python interfaces.

Focusing on common data preparation tasks for analytics and data science, RAPIDS offers a GPU-accelerated DataFrame that mimics the pandas API and is built on Apache Arrow. It integrates with scikit-learn and a variety of machine learning algorithms to maximize interoperability and performance without paying typical serialization costs. This allows acceleration for end-to-end pipelines—from data prep to machine learning to deep learning.  RAPIDS also includes support for multi-node, multi-GPU deployments, enabling vastly accelerated processing and training on much larger dataset sizes.