5 Best Ways to Convert CSV to Parquet in Python

by

💡 Problem Formulation: Converting CSV files to Parquet format is a common requirement for developers dealing with large data sets, as Parquet is optimized for size and speed of access. This article will guide you through various methods for performing this conversion in Python, starting from a CSV input like data.csv and resulting in a Parquet output data.parquet.

Method 1: Using Pandas with pyarrow

Pandas is a powerful data manipulation library for Python featuring an easy-to-use API. pyarrow is an Apache Arrow-based library to interact with columnar data efficiently. Together, they can convert CSV to Parquet effectively with compression support.

Here’s an example:

import pandas as pd
df = pd.read_csv('data.csv')
df.to_parquet('data.parquet', engine='pyarrow')

Output: A Parquet file named data.parquet will be created in the working directory.

This code snippet reads the CSV file using Pandas’ read_csv() function, and writes it to a Parquet file using the to_parquet() function, with pyarrow as the underlying engine for the conversion. The advantage of using Pandas with pyarrow is the ease of use and powerful data manipulation features provided by Pandas.

Method 2: Using Dask

Dask is a parallel computing library that scales to larger-than-memory computations. When handling large data sets, Dask can be used to read a CSV file and convert it to Parquet without using a lot of memory.

Here’s an example:

import dask.dataframe as dd
ddf = dd.read_csv('data.csv')
ddf.to_parquet('data.parquet')

Output: A Parquet file named data.parquet will be created, possibly in multiple parts if the dataset is large.

The code reads the CSV using Dask’s dataframe, which is similar to Pandas but handles operations in a lazy manner, to preserve memory. It then writes the Parquet file in a distributed fashion. Dask is particularly useful when dealing with very large data sets.

Method 3: Fastparquet Library

Fastparquet is a Python implementation of the Parquet format, providing Pythonic API for reading and writing Parquet files along with various filtering and indexing capabilities.

Here’s an example:

import pandas as pd
import fastparquet as fp
df = pd.read_csv('data.csv')
fp.write('data.parquet', df)

Output: A Parquet file data.parquet is saved to disk.

This example demonstrates reading a CSV file into a Pandas dataframe and then using Fastparquet’s write() function to write the dataframe to a Parquet file. Fastparquet is particularly useful if you need more control over the Parquet file formatting and compression.

Method 4: Using PySpark

PySpark is the Python API for Apache Spark, a powerful distributed computing system. PySpark can be used to handle big data sets and convert between different file formats, including CSV to Parquet.

Here’s an example:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('csv_to_parquet').getOrCreate()
df = spark.read.csv('data.csv', header=True, inferSchema=True)
df.write.parquet('data.parquet')

Output: A Parquet file data.parquet will be created in the Spark’s specified output directory.

This code initiates a PySpark session, reads the CSV into a dataframe with schema inference and headers, and writes out the data in Parquet format. This method is instrumental when processing large data sets in distributed systems.

Bonus One-Liner Method 5: pandas_to_parquet with one-liner

You can utilize the ‘pandas_to_parquet’ library which is capable of converting a CSV file to Parquet format in a single line of code.

Here’s an example:

import pandas_to_parquet
pandas_to_parquet.csv_to_parquet('data.csv', 'data.parquet')

Output: Generates a Parquet file data.parquet in the current directory.

This library wraps the functionality of Pandas and pyarrow to provide a condensed and intuitive way to achieve the CSV to Parquet conversion swiftly, with a straightforward one-liner command.

Summary/Discussion

  • Method 1: Pandas with pyarrow. Ideal for general use cases. Integrates easily with data analysis workflows. Limited when handling extremely large datasets.
  • Method 2: Dask. Best for large data sets that do not fit in memory. Provides parallel computation. Has a steeper learning curve compared to Pandas.
  • Method 3: Fastparquet. Offers granular control over Parquet file features and compression. Good choice for specialized Parquet needs. Less user-friendly than Pandas.
  • Method 4: PySpark. Suitable for very large, distributed data sets. Requires a Spark cluster setup. Overkill for small to medium-sized datasets.
  • Bonus Method 5: pandas_to_parquet. Excellent for simple and rapid conversions. Lacks the flexibility offered by using Pandas directly.