5 Best Ways to Fill Missing Values in a DataFrame with Python

by

💡 Problem Formulation: Dataframes often contain missing values, which can disrupt statistical analyses and machine learning models. Python offers various methods to deal with such missing values. Imagine you have a DataFrame with various data types and columns – some numeric, others categorical. The desired output is a DataFrame where all missing values are handled appropriately in a manner befitting their data types.

Method 1: Fill with a Specific Value

The fillna() method can be used to replace all NaN elements with a given value. This is often used when the missing data has a reasonable default value, like zero for numerical data or a category such as ‘Unknown’ for categorical data. Specify the value inside the fillna() function to seamlessly handle the missing data.

Here’s an example:

import pandas as pd

# Creating a sample DataFrame with missing values
df = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': ['a', None, 'c', 'd']
})

# Filling missing values with a specified value
df_filled = df.fillna(0)
print(df_filled)

Output:

     A  B
0  1.0  a
1  2.0  0
2  0.0  c
3  4.0  d

This code snippet creates a simple DataFrame with missing values and uses fillna(0) to replace all NaN entries with 0. In column ‘A’, the numeric NaN is replaced with 0, and in column ‘B’, the categorical NaN is replaced with the string ‘0’.

Method 2: Forward Filling

Forward filling is a method that propagates the last observed non-null value down to the next null occurrence. The fillna(method='ffill') function performs this operation, and it is especially useful in time series datasets where the last known value is a good approximation for the next one.

Here’s an example:

df_forward_filled = df.fillna(method='ffill')
print(df_forward_filled)

Output:

     A  B
0  1.0  a
1  2.0  a
2  2.0  c
3  4.0  d

In this example, the forward fill method carries the last valid observation ‘2’ and ‘a’ forward to replace NaN. This is particularly useful if the data is sorted and the previous value is relevant for the next.

Method 3: Backward Filling

Conversely, backward filling takes the next non-null value and fills the current null value with it. The fillna(method='bfill') function is applied to achieve this. It’s most effective when future values carry information that can be used for past data points.

Here’s an example:

df_backward_filled = df.fillna(method='bfill')
print(df_backward_filled)

Output:

     A  B
0  1.0  a
1  2.0  c
2  4.0  c
3  4.0  d

This code fills the NaN in column ‘A’ with the next value ‘4’, and for column ‘B’, it fills with the next valid entry ‘c’. Backward filling is again beneficial for time series where future recordings can estimate the missing past data.

Method 4: Interpolation

Interpolation is a technique used to estimate missing values based on other existing values. The interpolate() method in pandas is versatile and can fill missing values in a variety of ways, including linear or polynomial methods. This is particularly useful for numerical data and time series.

Here’s an example:

df_interpolated = df.interpolate()
print(df_interpolated)

Output:

     A    B
0  1.0    a
1  2.0  NaN
2  3.0    c
3  4.0    d

The code snippet shows interpolation of the numeric column ‘A’ where the missing value is calculated to be the midpoint between ‘2’ and ‘4’, resulting in ‘3’. It is not applied to non-numeric data as seen in column ‘B’.

Bonus One-Liner Method 5: Fill with Mode or Median

Filling missing values with the mode or median can be a quick one-liner solution, especially for dealing with numerical outliers or categorical columns with a clear majority value. Utilize fillna() with df.mode() or df.median() to apply these methods.

Here’s an example:

df_filled_mode = df.fillna(df.mode().iloc[0])
print(df_filled_mode)

Output:

     A  B
0  1.0  a
1  2.0  a
2  1.0  c
3  4.0  d

This example demonstrates filling missing values by calculating the mode of each column. The mode of ‘A’ is 1 and ‘B’ is ‘a’, which are used to fill in the NaNs.

Summary/Discussion

  • Method 1: Fill with a specific value. Strengths: Simple and fast. Weaknesses: May not be suitable for all data types or datasets with no obvious default value.
  • Method 2: Forward filling. Strengths: Good for time-series data. Weaknesses: Assumes chronological order and that previous value is a suitable replacement.
  • Method 3: Backward filling. Strengths: Like forward filling, suitable for time-series data. Weaknesses: Relies on the assumption that the future value is a valid proxy for the missing data.
  • Method 4: Interpolation. Strengths: Offers a more nuanced approach for numerical data. Weaknesses: More complex and not applicable to categorical data.
  • Bonus Method 5: Fill with Mode or Median. Strengths: Effective for categorical data or skewed numerical data. Weaknesses: May introduce bias if the mode or median is not representative of the entire dataset.