Pandas DataFrame drop_duplicates() Method – Be on the Right Side of Change

Preparation

Before any data manipulation can occur, two (2) new libraries will require installation.

The Pandas library enables access to/from a DataFrame.
The NumPy library supports multi-dimensional arrays and matrices in addition to a collection of mathematical functions.

To install these libraries, navigate to an IDE terminal. At the command prompt ($), execute the code below. For the terminal used in this example, the command prompt is a dollar sign ($). Your terminal prompt may be different.

$ pip install pandas

Hit the <Enter> key on the keyboard to start the installation process.

$ pip install numpy

Hit the <Enter> key on the keyboard to start the installation process.

If the installations were successful, a message displays in the terminal indicating the same.

Feel free to view the PyCharm installation guide for the required libraries.

Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

import pandas as pd
import numpy as np

DataFrame drop_duplicates()

The drop_duplicates() method returns a DataFrame/Series with duplicate rows removed.

The syntax for this method is as follows:

DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)

Parameter	Description
`subset`	Specify the column(s) to locate the duplicates. By default, all columns.
`keep`	Determines what duplicates to keep, first or the last occurrence. By default, first.
`inplace`	If False creates a copy of the DataFrame/Series. By default, `False`. If True, the original DataFrame/Series updates.
`ignore_index`	If `True`, the returning axis will start the numbers from 0 – n value. By default, `False`.

For this example, Rivers Clothing has found a duplicate clothing line in the DataFrame. They need this duplicate removed. To perform this task, run the following code:

df = pd.DataFrame({'Tops':    [10.12, 12.23, 13.95],
                   'Tanks':   [11.35, 13.45, 14.98],
                   'Sweats':  [27.15, 21.85, 35.75],
                   'Pants':   [21.37, 56.99, 94.87],
                   'Sweats':  [27.15, 21.85, 35.75]})

result = df.drop_duplicates()
print(result)

Line [1] creates a DataFrame from a Dictionary of Lists and saves it to df.
Line [2] removes the duplicate row.
Line [3] outputs the result to the terminal.

Output

	Tops	Tanks	Sweats	Pants
0	10.12	11.35	27.15	21.37
1	12.23	13.45	21.85	56.99
2	13.95	14.98	35.75	94.87

More Pandas DataFrame Methods

Feel free to learn more about the previous and next pandas DataFrame methods (alphabetically) here:

👈 df.diff()

df.drop_level() 👉

Also, check out the full cheat sheet overview of all Pandas DataFrame methods.