5 Best Ways to Remove Duplicates from NumPy Arrays

by

💡 Problem Formulation: When working with NumPy arrays, it’s common to encounter duplicate values within your datasets. For various applications, such as data preprocessing and feature engineering, it’s essential to remove these duplicates to maintain data integrity and performance. Suppose you have an input NumPy array [1, 2, 2, 3, 3, 3, 4], and you want to obtain an output array of unique values [1, 2, 3, 4]. This article discusses the top methods to achieve this in Python using NumPy.

Method 1: Using numpy.unique()

This method involves the use of the numpy.unique() function, which returns the sorted unique elements of an array. It’s the most straightforward approach to remove duplicates.

Here’s an example:

import numpy as np

array_with_duplicates = np.array([1, 2, 2, 3, 3, 3, 4])
unique_array = np.unique(array_with_duplicates)

print(unique_array)

Output:

[1 2 3 4]

This code creates a NumPy array with duplicates, then utilizes the np.unique() function to find and return a new array with only the unique values, which are naturally sorted.

Method 2: Using Boolean Indexing

Boolean indexing in NumPy can be used to filter out non-unique elements. This method requires a bit more coding but offers flexibility if there’s a need to customize the uniqueness condition.

Here’s an example:

import numpy as np

data = np.array([1, 2, 2, 3, 3, 3, 4])
unique_values = np.array([data[i] not in data[:i] for i in range(len(data))])
unique_array = data[unique_values]

print(unique_array)

Output:

[1 2 3 4]

The code snippet uses a comprehension to create a boolean mask, which is then applied to the original array to retrieve the unique values.

Method 3: Using a Set to Identify Unique Elements

Converting a NumPy array to a set is a quick way to identify unique elements, however, it loses the array structure and ordering. The result can be converted back to an array if needed.

Here’s an example:

import numpy as np

array_with_duplicates = np.array([1, 2, 2, 3, 3, 3, 4])
unique_set = set(array_with_duplicates)
unique_array = np.array(list(unique_set))

print(unique_array)

Output:

[1 2 3 4]

The provided code converts the NumPy array to a set to filter unique values, then the set is converted back to a list, and ultimately back to a NumPy array.

Method 4: Using Fancy Indexing

Fancy indexing is a NumPy technique whereby an array or list is used in place of an index. By creating an array of indices for which the condition of uniqueness is met, we can construct an array of unique values.

Here’s an example:

import numpy as np

data = np.array([1, 2, 2, 3, 3, 3, 4])
indices = np.sort(np.unique(data, return_index=True)[1])
unique_array = data[indices]

print(unique_array)

Output:

[1 2 3 4]

This example uses the np.unique() function with the return_index=True flag to get the indices of the unique elements and then selects these elements from the array.

Bonus One-Liner Method 5: Using numpy.lib.arraysetops.unique()

For a more behind-the-scenes approach, the numpy.lib.arraysetops.unique() function, which actually powers np.unique(), can be called directly for a oneliner solution. This is more of an “under the hood” method with the same outcome.

Here’s an example:

import numpy as np

array_with_duplicates = np.array([1, 2, 2, 3, 3, 3, 4])
unique_array = np.lib.arraysetops.unique(array_with_duplicates)

print(unique_array)

Output:

[1 2 3 4]

Here, we directly invoke the function responsible for finding unique elements, which is typically accessed through np.unique(), to reach the same conclusion.

Summary/Discussion

  • Method 1: Using numpy.unique(). It is very efficient and simple to use. Cannot maintain the original array order if it’s important.
  • Method 2: Using Boolean Indexing. Offers more control and is good for customized conditions. It’s less efficient and more complex than other methods.
  • Method 3: Using a Set to Identify Unique Elements. It’s a quick solution but does not preserve order or array structure without additional steps.
  • Method 4: Using Fancy Indexing. Retains the possibility of preserving the original array’s order. It’s a bit more complex and not as intuitive.
  • Bonus Method 5: One-Liner Using numpy.lib.arraysetops.unique(). Allows direct access to the underlying unique function for the same outcome as np.unique(), which can be satisfying for those interested in what happens under the hood.