5 Effective Ways to Iterate Over Pandas DataFrame Columns

by

💡 Problem Formulation: When working with data in Pandas, a common task is to iterate over DataFrame columns to perform operations on each column individually. This could include tasks such as data cleaning, transformation, aggregation, or to extract information. For example, given a DataFrame with columns ‘A’, ‘B’, and ‘C’, you might want to apply a function to each column and output a dictionary or a Series with column names as keys and the results of the functions as values.

Method 1: Using iteritems()

This method involves the iteritems() function, which is a generator that yields pairs of column names and corresponding series for each column in the DataFrame. This is easy to use for iterating over columns, especially when you need both the column name and the data.

♥️ Info: Are you AI curious but you still have to create real impactful projects? Join our official AI builder club on Skool (only $5): SHIP! - One Project Per Month

Here’s an example:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6],
    'C': [7, 8, 9]
})

for label, content in df.iteritems():
    print(f'Column {label} has mean: {content.mean()}')

Output:

Column A has mean: 2.0
Column B has mean: 5.0
Column C has mean: 8.0

This snippet iterates through each column in the DataFrame, computes the mean of each column, and prints out the result with the column label.

Method 2: Using itertuples()

The itertuples() function is used to iterate over DataFrame rows as namedtuples, and it’s a fast and memory-efficient way to iterate over the rows. When you want to consider row-wise operations but still access the columns individually, this is the way to go.

Here’s an example:

for row in df.itertuples():
    print(row.A, row.B, row.C)

Output:

1 4 7
2 5 8
3 6 9

In this example, iterating over rows with itertuples() allows you to address each element by their column label within the row.

Method 3: Using apply()

The apply() function is used to apply a function along the input axis of the DataFrame. This function is versatile and can be used to iterate over columns, to apply the same function to each column.

Here’s an example:

df.apply(lambda x: x * 2)

Output:

   A   B   C
0  2   8  14
1  4  10  16
2  6  12  18

In this code, apply() is used with a lambda function to double each value in the DataFrame across all columns.

Method 4: Using Column Attributes

Column attributes enable you to access a column directly by its name as if it were an attribute of the DataFrame. This method is straightforward when dealing with column names that are valid Python variable names.

Here’s an example:

for column in ['A', 'B', 'C']:
    print(df[column].sum())

Output:

6
15
24

This snippet shows how to iterate over the list of column names and accessing each column directly to compute the sum of each column.

Bonus One-Liner Method 5: Using list comprehension

Python’s list comprehension is a concise way to iterate over columns. You can perform operations in a single line, but it’s less readable for complex operations.

Here’s an example:

[print(df[col].sum()) for col in df.columns]

Output:

6
15
24

The list comprehension creates a list of sum of each column, and here it’s used for printing the sum directly.

Summary/Discussion

  • Method 1: iteritems(). Best for when you need column names during iteration. Can be slower compared to other methods when working with large DataFrames.
  • Method 2: itertuples(). Fast and efficient for row-wise operations but can be used to iterate over columns in a roundabout way. Not as direct or intuitive for pure column-based operations.
  • Method 3: apply(). Versatile method that can iterate over any axis; however, it may not always be the most efficient in terms of performance.
  • Method 4: Column Attributes. Simple and Pythonic for accessing columns by their names, yet isn’t flexible with column names that aren’t valid Python variable names.
  • Method 5: List Comprehension. Quick and concise for simple operations, but can hinder readability for more complex iterations and isn’t useful when you want to iterate with side effects like printing.