Find and Drop Duplicates

In this lesson, we'll dive into methods of finding and dropping duplicated rows of a DataFrame.

This tutorial uses classic Iris dataset, which can be downloaded here Iris dataset.

1. Find Duplicated Rows via the duplicated() Function

The duplicated() function returns a boolean array, indicating the presence of duplicate rows within the dataset. This boolean array can be used as a filtering condition for getting duplicated rows removed from the original DataFrame.

By running code below, the empty result indicates that no duplicated rows pesent in the data.

To better illustrate how the functions works, we'll manually insert some duplicated rows to the data.

Insert duplicated rows

We will take the last 4 rows of data and append to the original dataset, resulting in duplicated rows within the new dataframe.

By running tail(8) function on the new dataframe df_duplicate, we can see that rows 150 to 153 are duplications of rows 146 to 149.

Now let's apply the duplicated() function on the new dataframe df_duplicate and see what it returns:

It identifies that the row 150 to 153 are duplications. By default, the duplicated() function identifies data as duplicates, excluding the first occurrence. In our example, it has flagged rows 150 to 153 as duplicates, where rows 146 to 149 represent the initial occurrences.

2. Drop Duplicated Rows via drop_duplicates() Function

Just like the duplicated() function identifies duplicate entries, the drop_duplicates() function eliminates all duplicate rows, keeping only the first occurrence.

As shown above, rows 150 to 153 have been removed by the drop_duplicates() function.

3. Find and Drop Duplicates Based on a Subset of Columns

In our earlier examples, we searched for duplicate rows with identical values in all columns. Yet, there are situations where we specifically need to identify duplicate rows within a subset of columns. To accomplish this, we can use the subset parameter of the duplicated() and drop_duplicates() functions.

Example below shows all rows having duplications in the Species column.

The duplicated() function looks for duplicates in the Species column. Since there are only 3 unique values in the Species column, it returns 147 rows of duplications out of 150 rows.

Next, we'll use drop_duplicates() to remove duplicated rows:

After removing duplicated species, we are left with only three rows with unique Species values. As mentioned previously, these three rows have been retained as they represent the initial occurrences.

Well done! We have examined functions for removing duplicate records. In our upcoming tutorial, we will embark on an exploration of data grouping methods.