5 Best Ways to Convert Python CSV to Nested Dict

💡 Problem Formulation: When working with CSV files in Python, developers often need to transform data into a nested dictionary for better accessibility and manipulation. This structure is particularly useful when dealing with complex data relationships. For example, inputs like CSV rows need to be converted into a nested dict where each key corresponds to some header, and the associated value is a dict of remaining headers to their respective row values.

Method 1: Using csv.DictReader and a Loop

The csv.DictReader utility reads the CSV file into dictionaries, allowing for an easy conversion into a nested dict by looping over the rows and nesting dictionaries based on designated keys.

Here’s an example:

import csv

def csv_to_nested_dict(filename):
    with open(filename, mode='r') as csvfile:
        reader = csv.DictReader(csvfile)
        nested_dict = {}
        for row in reader:
            key = row.pop('YourPrimaryKey')
            nested_dict[key] = row
    return nested_dict

nested = csv_to_nested_dict('yourfile.csv')

Output:

{
    'row1_key': {'column2': 'value1', 'column3': 'value2'},
    ...
}

This method uses csv.DictReader to convert each row of the CSV into a dictionary with the CSV header as keys. The primary key for nesting is removed from the row and used as the top-level key in our nested dictionary. Remaining column values continue to be associated with their respective headers in the nested dictionary.

Method 2: Pandas GroupBy

For those working in data analysis, Pandas offers powerful options including DataFrame.groupby to create nested dicts by grouping the data on a particular column.

Here’s an example:

import pandas as pd

def csv_to_nested_dict_pandas(filename):
    df = pd.read_csv(filename)
    return df.groupby('YourPrimaryKey').apply(lambda x: x.to_dict(orient='records')).to_dict()

nested = csv_to_nested_dict_pandas('yourfile.csv')

Output:

{
    'row1_key': [{'column2': 'value1', 'column3': 'value2'}],
    ...
}

The Pandas library is utilized here to read the CSV into a DataFrame, then the DataFrame is grouped by the primary key and converted to a dictionary for each group. The to_dict() method with orient='records' outputs a list of dict, making each value under the primary key a list containing a single dictionary.

Method 3: Using csv.reader with DefaultDict

This method combines Python’s built-in csv.reader and collections.defaultdict to automatically handle missing keys with a specified dictionary as the default value.

Here’s an example:

import csv
from collections import defaultdict

def csv_to_nested_dict_defaultdict(filename):
    with open(filename, mode='r') as csvfile:
        reader = csv.reader(csvfile)
        headers = next(reader)
        primary_key = headers[0]
        nested_dict = defaultdict(dict)
        for row in reader:
            nested_dict[row[0]].update(dict(zip(headers[1:], row[1:])))
    return dict(nested_dict)

nested = csv_to_nested_dict_defaultdict('yourfile.csv')

Output:

{
    'row1_key': {'column2': 'value1', 'column3': 'value2'},
    ...
}

In this approach, csv.reader reads the CSV file and the headers are extracted. A defaultdict is then used to create nested dictionaries without the need to check if the key exists beforehand. The update() and zip() functions are used to merge the column headers and corresponding values into the nested dictionary.

Method 4: Using Dict Comprehension

Dict comprehensions offer a concise way to create a nested dictionary by iterating over each row of the CSV reader object and nesting dictionaries in one line.

Here’s an example:

import csv

def csv_to_nested_dict_comprehension(filename):
    with open(filename, mode='r') as csvfile:
        reader = csv.reader(csvfile)
        headers = next(reader)
        nested_dict = {row[0]: dict(zip(headers[1:], row[1:])) for row in reader}
    return nested_dict

nested = csv_to_nested_dict_comprehension('yourfile.csv')

Output:

{
    'row1_key': {'column2': 'value1', 'column3': 'value2'},
    ...
}

Using dict comprehension, we create a nested dictionary wherein for each row of the CSV file, the first column’s value becomes the key, and the remaining values form the value dictionary. The zip() function pairs column headers with their respective row values to create the nested structure.

Bonus One-Liner Method 5: Using csv.DictReader and Dict Comprehension

This method combines csv.DictReader and dict comprehension to provide a one-liner solution for creating a nested dictionary from a CSV file.

Here’s an example:

import csv

with open('yourfile.csv', mode='r') as csvfile:
    nested_dict = {row['YourPrimaryKey']: row for row in csv.DictReader(csvfile)}

Output:

{
    'row1_key': {'YourPrimaryKey': 'row1_key', 'column2': 'value1', 'column3': 'value2'},
    ...
}

This concise one-liner utilizes csv.DictReader to read the CSV and then employs a dict comprehension to create a nested dictionary. The primary key from each row becomes the key, and the entire row dictionary (including the primary key) becomes the value.

Summary/Discussion

• Method 1: DictReader and Loop. Strengths: Straightforward and uses standard library only. Weaknesses: More verbose and requires manual handling of primary key.
• Method 2: Pandas GroupBy. Strengths: Suitable for large datasets and additional data manipulation. Weaknesses: Depends on external library and may be overkill for simple tasks.
• Method 3: csv.reader with DefaultDict. Strengths: Efficient handling of missing keys. Weaknesses: Slightly less readable due to explicit handling of the CSV structure.
• Method 4: Dict Comprehension. Strengths: Concise. Weaknesses: Can be less intuitive for those unfamiliar with comprehensions.
• Bonus One-Liner Method 5: DictReader with Comprehension. Strengths: Extremely concise one-liner. Weaknesses: Includes primary key in the nested value dictionary, which may be undesired.