­­Pandas DataFrame Missing Data Handling – backfill(), bfill(), fillna(), dropna(), and interpolate() – Part 11

Rate this post

The Pandas DataFrame/Series has several methods to handle Missing Data. When applied to a DataFrame/Series, these methods evaluate and modify the missing elements.

This is Part 11 of the DataFrame methods series:

  • Part 1 focuses on the DataFrame methods abs(), all(), any(), clip(), corr(), and corrwith().
  • Part 2 focuses on the DataFrame methods count(), cov(), cummax(), cummin(), cumprod(), cumsum().
  • Part 3 focuses on the DataFrame methods describe(), diff(), eval(), kurtosis().
  • Part 4 focuses on the DataFrame methods mad(), min(), max(), mean(), median(), and mode().
  • Part 5 focuses on the DataFrame methods pct_change(), quantile(), rank(), round(), prod(), and product().
  • Part 6 focuses on the DataFrame methods add_prefix(), add_suffix(), and align().
  • Part 7 focuses on the DataFrame methods at_time(), between_time(), drop(), drop_duplicates() and duplicated().
  • Part 8 focuses on the DataFrame methods equals(), filter(), first(), last(), head(), and tail()
  • Part 9 focuses on the DataFrame methods equals(), filter(), first(), last(), head(), and tail()
  • Part 10 focuses on the DataFrame methods reset_index(), sample(), set_axis(), set_index(), take(), and truncate()
  • Part 11 focuses on the DataFrame methods backfill(), bfill(), fillna(), dropna(), and interpolate()
  • Part 12 focuses on the DataFrame methods isna(), isnull(), notna(), notnull(), pad() and replace()
  • Part 13 focuses on the DataFrame methods drop_level(), pivot(), pivot_table(), reorder_levels(), sort_values() and sort_index()
  • Part 14 focuses on the DataFrame methods nlargest(), nsmallest(), swap_level(), stack(), unstack() and swap_axes()
  • Part 15 focuses on the DataFrame methods melt(), explode(), squeeze(), to_xarray(), t() and transpose()
  • Part 16 focuses on the DataFrame methods append(), assign(), compare(), join(), merge() and update()
  • Part 17 focuses on the DataFrame methods asfreq(), asof(), shift(), slice_shift(), tshift(), first_valid_index(), and last_valid_index()
  • Part 18 focuses on the DataFrame methods resample(), to_period(), to_timestamp(), tz_localize(), and tz_convert()
  • Part 19 focuses on the visualization aspect of DataFrames and Series via plotting, such as plot(), and plot.area().
  • Part 20 focuses on continuing the visualization aspect of DataFrames and Series via plotting such as hexbin, hist, pie, and scatter plots.
  • Part 21 focuses on converting one data type to/from another data type.
  • Part 22 focuses on converting one data type to/from another data type.
  • Part 23 focuses on converting one data type to/from another data type.
  • Part 24 focuses on converting one data type to/from another data type.
  • Part 25 focuses on converting one data type to/from another data type.

Preparation

Before any data manipulation can occur, two (2) new libraries will require installation.

  • The Pandas library enables access to/from a DataFrame.
  • The NumPy library supports multi-dimensional arrays and matrices in addition to a collection of mathematical functions.

To install these libraries, navigate to an IDE terminal. At the command prompt ($), execute the code below. For the terminal used in this example, the command prompt is a dollar sign ($). Your terminal prompt may be different.

$ pip install pandas

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install numpy

Hit the <Enter> key on the keyboard to start the installation process.

If the installations were successful, a message displays in the terminal indicating the same.


Feel free to view the PyCharm installation guide for the required libraries.


Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

import pandas as pd
import numpy as np 

DataFrame backfill() and bfill()

The DataFrame backfill() and bfill() methods backward fill missing data (such as np.nan, None, NaN, and NaT values) from the DataFrame/Series.

The syntax for these methods is as follows:

DataFrame.backfill(axis=None, inplace=False, limit=None, downcast=None)
DataFrame.bfill(axis=None, inplace=False, limit=None, downcast=None)
axisIf zero (0) or index is selected, apply to each column. Default 0.
If one (1) apply to each row.
inplaceIf set to True, the changes apply to the original DataFrame/Series. If False, the changes apply to a new DataFrame/Series. By default, False.
limitThe maximum number of elements to backward fill.
downcastThe only available selection is infer. This parameter attempts to convert floats (float64) to integers (int64).

Throughout this article, we use the same DataFrame example. This DataFrame contains three (3) rows with missing data. Each example attempts to handle the missing data.

In this example, the DataFrame contains some missing data. This code will attempt to (replace) these values using the bfill() method.

Code – Example 1

df = pd.DataFrame({'Data-1':  [None, 11, 12], 
                   'Data-2':  [13, 14, None],
                   'Data-3':  [None, 15, 16]})
print(df)

result = df.bfill(axis='rows')
print(result)
  • Line [1] creates a dictionary of lists and saves it to df.
  • Line [2] outputs the DataFrame to the terminal. The missing values convert to NaN.
  • Line [3] backfills the NaN values across the rows. This output saves to the result variable.
  • Line [4] outputs the result to the terminal.

Output

df
 Data-1 Data-2 Data-3
0NaN13.0NaN
111.014.015.0
212.0NaN16.0
result
 Data-1 Data-2 Data-3
011.013.015.0
111.014.015.0
212.0NaN16.0

💡 Notebackfill/bfill tries to fill in the NaN values with data from the same position in the next row. If there is no next row or the next row contains NaN, the value does not change.

Code – Example 2

df = pd.DataFrame({'Data-1':  [None, 11, 12], 
                   'Data-2':  [13, 14, 'NaN'],
                   'Data-3':  [None, 15, 16]})
print(df)

result = df.bfill(axis='rows')
print(result)
  • Line [1] creates a dictionary of lists and saves it to df.
  • Line [2] outputs the DataFrame to the terminal. The missing values convert to NaN.
  • Line [3] backfills the NaN values across the rows. This output saves to the result variable.
  • Line [4] outputs the result to the terminal.

Output

df
 Data-1 Data-2 Data-3
0NaN13.0NaN
111.014.015.0
212.0NaN16.0
result
 Data-1 Data-2 Data-3
011.013.015.0
111.014.015.0
212.0NaN16.0

💡Note: The output is identical to that in Example 1.

Code – Example 3

df = pd.DataFrame({'Data-1':  [None, 11, 12], 
                   'Data-2':  [13, 14, 'NaN'],
                   'Data-3':  [None, 15, 16]})
print(df)

result = df.bfill(axis='rows', downcast='infer')
print(result)
  • Line [1] creates a dictionary of lists and saves it to df.
  • Line [2] outputs the DataFrame to the terminal. All missing values convert to NaN.
  • Line [3] backfills the NaN values across the rows. The infer parameter attempts to change the dtype across the DataFrame/Series. This output saves to the result variable.
  • Line [4] outputs the result to the terminal.

Output

df
 Data-1 Data-2 Data-3
0NaN13.0NaN
111.014.015.0
212.0NaN16.0
result
 Data-1 Data-2 Data-3
0111315
1111415
212NaN16

DataFrame fillna()

The fillna() method fills in the DataFrame/Series missing data (NaN/None) with the content of the value parameter is shown below.

The syntax for this method is as follows:

Frame.fillna(value=None, method=None, axis=None, 
             inplace=False, limit=None, downcast=None)
valueThis value is a value to fill in the missing values. This value can be a single value or a dictionary for a value-for-value replacement. Anything not in the dictionary remains unchanged.
methodThe method to use to fill in the missing values. The choices are: pad/ffill: complete with last value. backfill/bfill: complete with next value.
axisIf zero (0) or index is selected, apply to each column. Default 0.
If one (1) apply to each row.
inplaceIf set to True, the changes apply to the original DataFrame/Series. If False, the changes apply to a new DataFrame/Series. By default, False.
limitThe maximum number of elements to backward/forward fill.
downcastThe only available selection is the infer option. This attempts to convert floats (float64) to integers (int64).

In this example, the DataFrame contains some missing data. This code will attempt to (replace) these values using the fillna() method.

df = pd.DataFrame({'Data-1':  [np.nan, 11, 12], 
                   'Data-2':  [13, 14, np.nan],
                   'Data-3':  [np.nan, 15, 16]},
                   index=['Row-1', 'Row-2', 'Row-3'])
print(df)

result = df.fillna(22, downcast='infer')
print(result)
  • Line [1] creates a dictionary of lists and saves it to df.
  • Line [2] outputs the DataFrame to the terminal. All np.nan values convert to NaN.
  • Line [3] fills in the missing values across the rows with the value 22. The infer parameter attempts to change the dtype across the DataFrame/Series. This output saves to the result variable.
  • Line [4] outputs the result to the terminal.

Output

df
 Data-1 Data-2 Data-3
Row-1    NaN13.0NaN
Row-2   11.014.015.0
Row-3   12.0NaN16.0
result
 Data-1 Data-2 Data-3
Row-1    221322
Row-2   111415
Row-3   122216

💡 Note: The output using ffill() is the same as if you use fillna() and pass the method parameter as ffill.


DataFrame dropna()

The dropna() method removes missing data from a DataFrame/Series.

The syntax for this method is as follows:

DataFrame.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)
axisIf zero (0) or index is selected, apply to each column. Default 0.
If one (1) apply to each row.
howDetermines when a row/column removes from the DataFrame. The available options are: Any: if any NA values, drop row/column. All: if all NA values, then drop row/column.
threshThis parameter requires that there many Non-NA values.
subsetThis subset is the label(s) along the other axis to include. Must be in an array-like format and contain a list of columns in the subset.
inplaceIf set to True, the changes apply to the original DataFrame/Series. If False, the changes apply to a new DataFrame/Series. By default, False.

Note: A list of a few possible empty values are:

  • 'NaN'
  • pd.NaN
  • np.nan
  • None
  • NaT

In this example, the DataFrame contains some missing data. Therefore, this code will attempt to remove the rows that contain these values.

df = pd.DataFrame({'Data-1':  [np.nan, 11, 12], 
                   'Data-2':  [13, 14, pd.NaT],
                   'Data-3':  [None, 15, 16]},
                   index=['Row-1', 'Row-2', 'Row-3'])
print(df)

result = df.dropna()
print(result)
  • Line [1] creates a dictionary of lists and saves it to df.
  • Line [2] outputs the DataFrame to the terminal.
  • Line [3] removes the rows containing missing values. This output saves to the result variable.
  • Line [4] outputs the result to the terminal.

Output

df
 Data-1 Data-2 Data-3
Row-1    NaN13.0NaN
Row-2   11.014.015.0
Row-3   12.0NaT16.0
result
 Data-1 Data-2 Data-3
Row-2   11.014.015.0

💡 Note: Row-2 is the only row that contains valid data and the only row left after applying the dropna() method.


DataFrame interpolate()

The interpolate() method fills all NaN values using interpolation.

The syntax for this method is as follows:

DataFrame.interpolate(method='linear', axis=0, limit=None, inplace=False, limit_direction=None, limit_area=None, downcast=None, **kwargs)
methodThis parameter is the interpolation technique to use. The available options are:
linear: Ignore the index. Treat as spaced equally.
time:  This parameter works on daily/high res to interpolate a specified time interval.
index, values: Use the numeric values of the index.
pad: Fill in any NaN values with existing values.
nearest/zero/slinear/quadratic/cubic/spline/barycentric/polynomial: Use the numeric values of the index. Polynomial and spline need an order (int).
krogh/piecewise_polynomial/spline/pchip/akima/cubic/spline: Wraps around the SciPy Interpolation method(s) of similar name(s).
from_derivatives: Refers to scipy.interpolate.BPoly.from_derivatives which replaces ‘piecewise_polynomial’ interpolation method in Scipy 0.18.
axisIf zero (0) or index is selected, apply to each column. Default 0.
If one (1) apply to each row.
limitThe maximum number of successive NaN values to fill. Must be more than zero (0).
inplaceIf set to True, the changes apply to the original DataFrame/Series. If False, the changes apply to a new DataFrame/Series. By default, False.
limit_directionThe successive NaN values fill in with the specified direction.
– If limit: If method pad/ffill, set direction to forward. If method backfill/bfill, set direction to backward.
– If no limit: If method backfill/bfill, the default direction is backward. Otherwise forward.

The DataFrame in this example contains missing data. This code will attempt to replace these values.

df = pd.DataFrame({'Data-1':  [np.nan, 11, 12], 
                   'Data-2':  [13, 14, pd.NaT],
                   'Data-3':  [None, 15, 16]},
                   index=['Row-1', 'Row-2', 'Row-3'])
print(df)

result = df.interpolate(method='linear', limit_direction='backward', axis=0)
print(result)
  • Line [1] creates a dictionary of lists and saves it to df.
  • Line [2] outputs the DataFrame to the terminal.
  • Line [3] interpolates and sets the parameters to linear, the direction to backward, and the axis to zero (0). This output saves to the result variable.
  • Line [4] outputs the result to the terminal.

Output

df
 Data-1 Data-2 Data-3
Row-1    NaN13.0NaN
Row-2   11.014.015.0
Row-3   12.0NaT16.0
result
 Data-1 Data-2 Data-3
Row-1    11.013.015.0
Row-2   11.014.015.0
Row-3   12.0NaT16.0