5 Best Ways to Fill Missing Column Values in Pandas with Constant

by

💡 Problem Formulation: When handling datasets with Python’s pandas library, dealing with missing values can be inevitable. Missing values are usually represented by NaN (not a number) and can impede various data analysis processes. This article illustrates how to effectively fill these missing column values with a constant, showcasing input data with NaNs and the desired output with filled values.

Method 1: Using fillna()

One of the most straightforward methods to fill missing values is using the fillna() function from the pandas library. This function allows you to fill NaN values with a specified constant across the DataFrame or in selected columns.

Here’s an example:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [None, 2, 3, 4]
})

df.fillna(0, inplace=True)
print(df)

The output is:

     A    B
0  1.0  0.0
1  2.0  2.0
2  0.0  3.0
3  4.0  4.0

This code snippet creates a DataFrame with missing values and uses fillna() with the argument 0 to replace all NaN values with 0. The operation is done in-place to modify the existing DataFrame.

Method 2: Assigning Directly to DataFrame Columns

If only specific columns have missing values that you want to fill, directly assigning a constant to these columns is a quick and easy solution. This method is particularly useful when dealing with many columns and you only want to apply the change to a subset.

Here’s an example:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [None, 2, 3, 4]
})

df['A'] = df['A'].fillna(0)
print(df)

The output is:

     A    B
0  1.0  NaN
1  2.0  2.0
2  0.0  3.0
3  4.0  4.0

Here, only the ‘A’ column is filled with the constant 0 using the fillna() method. The ‘B’ column remains unchanged, leaving its NaN values intact.

Method 3: Using the apply() Function

The apply() function can be used to fill missing values across an entire DataFrame or within specific columns. It is a versatile method that can apply a function along an axis of the DataFrame, and is helpful when you need to apply more complex criteria for replacing NaN values.

Here’s an example:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [None, 2, 3, 4]
})

df = df.apply(lambda x: x.fillna(0))
print(df)

The output is:

     A    B
0  1.0  0.0
1  2.0  2.0
2  0.0  3.0
3  4.0  4.0

In this code snippet, we apply a lambda function that calls fillna() on each column of the DataFrame, filling all NaN values with the constant 0.

Method 4: Using replace() to Fill NaN

The replace() method is a powerful tool for replacing various values within a DataFrame. While typically used for replacing a range of values, it can also replace NaN values when they are identified as such.

Here’s an example:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [None, 2, 3, 4]
})

df.replace(to_replace=[None], value=0, inplace=True)
print(df)

The output is:

     A    B
0  1.0  0.0
1  2.0  2.0
2  0.0  3.0
3  4.0  4.0

This code replaces all occurrences of None, which pandas interprets as NaN, with 0. The inplace=True argument makes the changes persist in the original DataFrame.

Bonus One-Liner Method 5 Using where()

The where() function comes in handy for condition-based replacements, including filling NaN values. It allows for inline conditional logic, making it a concise method for dealing with missing data.

Here’s an example:

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [None, 2, 3, 4]
})

df = df.where(pd.notna(df), other=0)
print(df)

The output is:

     A    B
0  1.0  0.0
1  2.0  2.0
2  0.0  3.0
3  4.0  4.0

This elegant one-liner uses where() to keep the original values when they are not NaN and to replace NaN values with 0.

Summary/Discussion

  • Method 1: fillna(): Simple and direct. Works well for filling NaN values across the entire DataFrame. The method does not allow for conditional replacements which can be limiting in certain scenarios.
  • Method 2: Column Assignment: Best for targeting specific columns. Straightforward but requires extra steps if you wish to target multiple specific columns individually.
  • Method 3: apply() Function: Offers flexibility with a functional approach, allowing for complex functions to be applied. It can be less efficient for larger datasets due to the overhead of function calls.
  • Method 4: replace(): Highly versatile for a wider range of value replacement. Good for handling different types of data. May be overkill when only filling NaN values.
  • Method 5: where(): Concise and powerful for condition-based operations. It does, however, require a good understanding of conditional statements within pandas.