5 Best Ways to Check for Non-null Values with Python Pandas Series

by

💡 Problem Formulation: When working with data in Python using the pandas library, it’s common to need to filter out null or missing values. The notnull() method in pandas Series is a crucial tool for this task. Suppose you have a pandas Series with some null values and you want to identify all non-null elements—the output should be a boolean series that can correctly indicate the presence of non-null values.

Method 1: Using notnull() to Filter Non-null Entries

This method uses the built-in pandas Series method notnull(), which returns a Boolean Series indicating whether each value is not null. It’s particularly handy for filtering out null values in subsequent operations. The function format is Series.notnull().

Here’s an example:

import pandas as pd

s = pd.Series([1, None, 3, None, 5])
non_nulls = s.notnull()

print(non_nulls)

Output:

0     True
1    False
2     True
3    False
4     True
dtype: bool

In this snippet, the notnull() method is applied to a pandas Series s, creating a boolean mask, non_nulls, where each true value corresponds to a non-null entry in the original series. This boolean Series can then be used for indexing or filtering purposes.

Method 2: Using Boolean Indexing with notnull()

Boolean indexing in pandas allows us to use a boolean vector to filter the data. Applying notnull() within a bracket operation filters out all null values, providing a filtered Series with non-null values only. It is written as Series[Series.notnull()].

Here’s an example:

import pandas as pd

s = pd.Series([2, None, 4, None, 6])
filtered_s = s[s.notnull()]

print(filtered_s)

Output:

0    2.0
2    4.0
4    6.0
dtype: float64

Here, the s.notnull() method creates a boolean Series. This boolean Series is then used to index the original Series s, resulting in filtered_s, which contains only the non-null entries from s.

Method 3: Combining notnull() with Other Methods for Conditional Operations

It is often useful to combine notnull() with other pandas methods for conditional manipulations. For instance, using notnull() together with the loc property enables us to selectively apply operations to non-null entries. The approach can be formatted as Series.loc[Series.notnull()].

Here’s an example:

import pandas as pd

s = pd.Series([10, None, 20, None, 30])
s.loc[s.notnull()] *= 10

print(s)

Output:

0    100.0
1      NaN
2    200.0
3      NaN
4    300.0
dtype: float64

The example demonstrates how to perform operations conditionally on non-null values of a Series. Here, the notnull() method is used in conjunction with loc to multiply only the non-null elements by 10.

Method 4: Using dropna() to Exclude Null Values

Alternative to filtering with notnull(), pandas provides the dropna() method, which returns a new Series with null values removed. Unlike notnull(), which returns a boolean mask, dropna() gives us the filtered Series directly and can be used as Series.dropna().

Here’s an example:

import pandas as pd

s = pd.Series([3, None, 6, None, 9])
non_null_series = s.dropna()

print(non_null_series)

Output:

0    3.0
2    6.0
4    9.0
dtype: float64

By using dropna(), you obtain a Series non_null_series that only contains the original values of s which were not null, effectively excluding any NaN or None values.

Bonus One-Liner Method 5: Using a Lambda Function with apply()

For custom filtering or when integrating notnull() into complex operations, apply a lambda function using apply(). This allows for inline anonymous functions that expand the use of notnull(). It is used as: Series.apply(lambda x: x is not None).

Here’s an example:

import pandas as pd

s = pd.Series([7, None, 13, None, 21])
non_nulls = s.apply(lambda x: x is not None)

print(non_nulls)

Output:

0     True
1    False
2     True
3    False
4     True
dtype: bool

In this one-liner, the lambda function checks each element of the Series s for the condition that the element is not None, which mirrors the functionality of notnull(), and can be integrated into more complex functions if necessary.

Summary/Discussion

    Method 1: notnull(). Straightforward and widely used. Best for obtaining a boolean mask. Not a direct way to obtain the filtered data. Method 2: Boolean Indexing with notnull(). Direct and concise. Filters non-null values effectively. Requires knowledge of boolean indexing. Method 3: Combine notnull() with Other Methods. Flexible and powerful in conditional data manipulation. Slightly more complex syntax. Method 4: dropna(). Simple and returns a Series directly. Does not provide a boolean mask for further operations. Method 5: Lambda Function with apply(). Highly customizable and can be integrated into complex functions. Potentially overkill for simple non-null checks.