Find and drop duplicate rows of a data frame — base R and dplyr, same result
Find and remove duplicate rows in R with base-R duplicated() and unique(), or dplyr distinct(). Remove duplicates across all columns or by a single key column, keep the first or last occurrence, and count how many duplicates you have — both engines, side by side.
June 26, 2026
July 7, 2026
duplicated(df) returns a TRUE/FALSE vector flagging each repeated row.unique(df) or df[!duplicated(df), ]; dplyr: distinct(df).df[!duplicated(df$id), ] (base R) or distinct(df, id, .keep_all = TRUE) (dplyr).duplicated() keeps the first occurrence; pass fromLast = TRUE to keep the last instead.You merged two data files, scraped a table, or appended new records, and now some rows appear more than once. Duplicate rows inflate counts, bias summaries, and break joins, so removing them is a routine first cleanup step.
There are two routes, and you’ll see both in real code:
duplicated() flags repeats, unique() drops them; no package needed.distinct() keeps the unique rows in one readable call.This lesson shows both, side by side, for finding duplicates, removing them across all columns, removing them by a key column, and choosing which copy to keep. The two give the same result.
We’ll build a small data frame with deliberate duplicates so the output is easy to follow:
id group score
1 1 a 10
2 2 b 20
3 2 b 20
4 3 c 30
5 3 c 30
6 3 d 40Rows 2–3 are identical, and so are rows 4–5. Row 6 shares id 3 but has a different group and score.
Before removing anything, look at what’s duplicated. duplicated() returns a logical vector: TRUE for every row that is a repeat of an earlier one.
The first time a row appears it’s FALSE; the second (and later) time it’s TRUE. Rows 3 and 5 are flagged because they repeat rows 2 and 4.
To see the duplicated rows, subset with that vector:
To count how many duplicates you have, sum the logical vector — TRUE counts as 1:
Two duplicate rows. table() is handy when you want the count per repeated value of a single column:
id 2 appears twice and id 3 appears three times.
A row counts as a duplicate when every column matches an earlier row.
unique() or !duplicated()unique() is the most direct tool: it returns the data frame with duplicate rows removed.
df[!duplicated(df), ] does the same thing — keep the rows that are not flagged as duplicates. The ! negates the logical vector:
Both drop rows 3 and 5 and keep the first copy of each. unique() is the shorthand; !duplicated() is the pattern to reach for when you want to dedupe by some of the columns (next section).
distinct()distinct() keeps the unique rows. With no column arguments it looks at all columns:
Same four rows. distinct() reads cleanly and slots into a pipeline.
Often “duplicate” means the same id, even if other columns differ. Here you choose which column(s) define a duplicate.
!duplicated() on the key columnApply duplicated() to the key column only, then subset:
This keeps the first row for each id — note that id 3 keeps row 4 (group c), and rows 5 and 6 are dropped even though row 6 had different values. Deduping by a key always throws away information from the dropped rows, so be sure the key is what you mean.
distinct(.keep_all = TRUE)Name the key column(s) in distinct(). By default distinct() returns only the columns you name, so add .keep_all = TRUE to keep all the other columns of the first matching row:
Same result as the base-R route — one row per id, keeping the first occurrence and all its columns.
By default both engines keep the first occurrence of each duplicate. To keep the last instead, flag duplicates from the end with fromLast = TRUE:
Now id 3 keeps row 6 (group d) — the last row for that id — instead of the first. Use this when the most recent record is the one you want to keep. In dplyr, sort the data so the row you want comes first, then distinct().
The blocks above ran at build time. Edit this one and press Run to remove duplicates on your own — it executes in your browser via webR (no install).
Paste a messy data frame with repeated records and ask Prova “remove the duplicate rows, keeping the most recent per customer” — it writes the distinct() or duplicated(fromLast = TRUE) call, and because the runtime is right here you run it to confirm the right rows survived on your data. Ask Prova →
Working in Python? A pandas drop_duplicates guide is coming to the Python series.
Duplicates aren’t removed even though rows “look” the same. A row is only a duplicate when every column matches exactly. Trailing spaces ("a" vs "a "), different capitalisation ("A" vs "a"), or a stray factor level all make rows distinct. Clean the offending column first (e.g. trimws(), tolower()), then dedupe.
Numbers that print the same but aren’t equal. Floating-point values computed different ways (0.1 + 0.2 vs 0.3) print identically but differ at the 16th decimal, so duplicated() treats them as distinct. Round the key column with round(x, 6) before deduping if “near-equal” should count as a duplicate.
distinct(df, id) dropped my other columns. That’s the default — distinct() with named columns returns only those columns. Add .keep_all = TRUE to keep the rest of each first matching row.
Use base-R unique(df) or df[!duplicated(df), ], or dplyr distinct(df). All three keep the first copy of each fully-duplicated row and drop the rest, giving the same result.
duplicated(df) returns a TRUE/FALSE vector flagging each repeated row. Subset with it — df[duplicated(df), ] — to see the duplicates, or sum(duplicated(df)) to count them.
Apply the test to that column: df[!duplicated(df$id), ] (base R) or distinct(df, id, .keep_all = TRUE) (dplyr). Both keep the first row for each value of the key column and all its other columns.
They produce the same rows for whole-row deduplication. unique() is base R and needs no package; distinct() is dplyr, reads clearly, and lets you dedupe by named columns with .keep_all = TRUE. Use whichever fits your code base.
Pass fromLast = TRUE to duplicated(): df[!duplicated(df$id, fromLast = TRUE), ] keeps the last row for each key. In dplyr, sort so the row you want comes first, then call distinct().
The data frame below has fully-duplicated rows and repeated id values. First remove only the rows that are duplicated across all columns (base R), then keep just one row per id with all columns (dplyr) — and check how many rows each leaves.
library(dplyr)
orders <- data.frame(
id = c(1, 1, 2, 2, 3),
item = c("pen", "pen", "ink", "pad", "pen"),
qty = c(2, 2, 1, 5, 3)
)
no_dupes <- orders[!duplicated(orders), ] # drops the duplicated pen row -> 4 rows
one_per_id <- distinct(orders, id, .keep_all = TRUE) # one row per id -> 3 rows
c(no_dupes = nrow(no_dupes), one_per_id = nrow(one_per_id))
library(dplyr)
orders <- data.frame(
id = c(1, 1, 2, 2, 3),
item = c("pen", "pen", "ink", "pad", "pen"),
qty = c(2, 2, 1, 5, 3)
)
no_dupes <- orders[!duplicated(orders), ] # drops the duplicated pen row -> 4 rows
one_per_id <- distinct(orders, id, .keep_all = TRUE) # one row per id -> 3 rows
c(no_dupes = nrow(no_dupes), one_per_id = nrow(one_per_id))You have a data frame df with a column id and want one row per id, keeping the last occurrence. Which is correct?
A. df[!duplicated(df$id), ] B. df[!duplicated(df$id, fromLast = TRUE), ] C. distinct(df, id)
Show answer
B. fromLast = TRUE flags duplicates from the bottom up, so the last row for each id is the one kept. A keeps the first occurrence; C keeps the first occurrence and drops every column except id (you’d need .keep_all = TRUE).
Removing duplicates comes down to two equivalent toolsets: base-R unique() / df[!duplicated(df), ] or dplyr distinct(). Use duplicated() to find and count repeats first, dedupe by a key column with df[!duplicated(df$key), ] or distinct(df, key, .keep_all = TRUE), and reach for fromLast = TRUE when the last record is the one to keep.
arrange()/order() (often the step before deduping). · Subset rows in R — filter observations by condition. · Select columns in R — keep or drop columns. · Rename columns in R — clean up column names.Prove you can do it. Master the whole Data Wrangling in R & Python series — track your path, build projects, and earn a certificate.
Go Pro — unlimited Prova on your own data and a verifiable certificate that proves the skill.
from $15/mo billed yearly
✓ You're Pro — keep going. The runtime is the judge.
Get new R & Python lessons by email
Practical, reproducible, no spam. Unsubscribe anytime.
Double opt-in. We never share your email.
This lesson is reproducible: every result on this page was produced by the code shown — edit any block and press Run to reproduce it in your browser. The runtime is the judge.
@online{2026,
author = {},
title = {Remove {Duplicates} in {R:} Duplicated, Unique \& Distinct},
date = {2026-06-26},
url = {https://www.datanovia.com/learn/programming/data-wrangling/remove-duplicates-in-r},
langid = {en}
}