5 Best Ways to Convert a List of Nested Dictionaries into a Pandas DataFrame in Python

Rate this post

πŸ’‘ Problem Formulation: Python developers often face the challenge of transforming data stored as a list of nested dictionaries into a structured format such as a Pandas DataFrame. The goal is to convert data like [{'A': {'a1': 1, 'a2': 2}, 'B': 3},{'A': {'a1': 4, 'a2': 5}, 'B': 6}] into a DataFrame that tabulates the nested information for further data analysis tasks.

Method 1: Using the pandas DataFrame Constructor

This method is the most straightforward: it involves passing the list of nested dictionaries directly to the DataFrame constructor provided by Pandas. The constructor is designed to accept various data formats, including nested dictionaries, and performs conversion internally. It automatically aligns data based on keys and handles missing values.

Here’s an example:

import pandas as pd

data = [{'A': {'a1': 1, 'a2': 2}, 'B': 3},
        {'A': {'a1': 4, 'a2': 5}, 'B': 6}]

df = pd.DataFrame(data)
print(df)

Output:

               A  B
0  {'a1': 1, 'a2': 2}  3
1  {'a1': 4, 'a2': 5}  6

This code snippet uses the DataFrame constructor from the Pandas library to convert a list of nested dictionaries into a DataFrame. The result is a DataFrame where each dictionary becomes a row, and nested dictionaries remain nested within cells.

Method 2: Pandas json_normalize

Pandas provides a function called json_normalize that can handle the conversion of nested dictionary structures into a flat table. This method is highly efficient when working with JSON data structures and can unpivot deeply nested dictionaries into a tabular form, creating separate columns for nested keys.

Here’s an example:

import pandas as pd
from pandas.io.json import json_normalize

data = [{'A': {'a1': 1, 'a2': 2}, 'B': 3},
        {'A': {'a1': 4, 'a2': 5}, 'B': 6}]

df = json_normalize(data)
print(df)

Output:

   B  A.a1  A.a2
0  3     1     2
1  6     4     5

With json_normalize, this snippet efficiently flattens the list of nested dictionaries into a DataFrame, creating separate columns for each nested key and pairing them with their parent’s keys using a delimiter.

Method 3: Using a For Loop and pd.concat

Sometimes, a more manual approach can offer greater control over the data transformation process. By iterating over the list of nested dictionaries with a for loop, each nested dictionary can be converted to a DataFrame and concatenated to a final DataFrame using pd.concat.

Here’s an example:

import pandas as pd

data = [{'A': {'a1': 1, 'a2': 2}, 'B': 3},
        {'A': {'a1': 4, 'a2': 5}, 'B': 6}]

frames = []
for d in data:
    frames.append(pd.DataFrame.from_dict({(i, j): d[i][j] 
                                          for i in d.keys() 
                                          for j in d[i].keys()},
                                          orient='index').T)
df = pd.concat(frames).reset_index(drop=True)
print(df)

Output:

   (A, a1)  (A, a2)  (B, )
0        1        2      3
1        4        5      6

This approach utilizes Python’s list comprehension and dictionary manipulation to flatten each nested dictionary and construct individual DataFrames. Subsequently, these DataFrames are merged into a single DataFrame with pd.concat.

Method 4: Recursive Unpacking with Custom Function

For deeply nested dictionary structures that might extend beyond one level, a custom recursive function can be written to unpack each level of nesting until it reaches primitives, then convert the resulting structure into a DataFrame. This is particularly useful for very complex or inconsistently structured data.

Here’s an example:

import pandas as pd

def unpack(nested_dict, parent_key=''):
    items = []
    for k, v in nested_dict.items():
        new_key = f'{parent_key}.{k}' if parent_key else k
        if isinstance(v, dict):
            items.extend(unpack(v, new_key).items())
        else:
            items.append((new_key, v))
    return dict(items)

data = [{'A': {'a1': 1, 'a2': 2}, 'B': 3},
        {'A': {'a1': 4, 'a2': {'a3': 5}}, 'B': 6}]

unpacked_data = [unpack(d) for d in data]
df = pd.DataFrame(unpacked_data)
print(df)

Output:

   A.a1  A.a2    B  A.a2.a3
0     1   2.0  3.0      NaN
1     4   NaN  6.0      5.0

The custom recursive function, unpack, digs through each level of nesting, accumulating keys as it delves deeper. By calling this function on each dictionary in the list, a flat structure is obtained, which is then fed into the DataFrame constructor.

Bonus One-Liner Method 5: Inline dict Unpacking with List Comprehension

Lastly, a more hands-on Pythonic method combines dict unpacking with list comprehension to quickly convert a list of nested dictionaries to a DataFrame. It’s concise, but the readability might be lower for complex nesting.

Here’s an example:

import pandas as pd

data = [{'A': {'a1': 1, 'a2': 2}, 'B': 3},
        {'A': {'a1': 4, 'a2': 5}, 'B': 6}]

df = pd.DataFrame([{**x['A'], 'B': x['B']} for x in data])
print(df)

Output:

   a1  a2  B
0   1   2  3
1   4   5  6

The list comprehension unpacks the ‘A’ dictionary’s items into the outer dictionary, flattening the structure. This generates a list of dictionaries without nested structures, ready to be directly passed to the DataFrame constructor.

Summary/Discussion

  • Method 1: DataFrame Constructor. Direct and simple. However, it does not flatten the structure.
  • Method 2: json_normalize. Efficiently flattens nested dictionaries. Might require additional handling for deeply nested or inconsistent data structures.
  • Method 3: For Loop and pd.concat. Offers more control over the transformation. More verbose and potentially slower for large datasets.
  • Method 4: Recursive Unpacking with Custom Function. Versatile for complex nesting. Can be overkill for simple or well-structured data, and may add unnecessary complexity.
  • Bonus Method 5: Inline dict Unpacking. Extremely concise. May not be suitable for deeply nested or varied structures, and can reduce code readability.