Comments on: Identify and Remove Duplicate Data in R

By: kassambara

kassambara — Fri, 22 May 2020 10:21:56 +0000

To overwrite, your original file, type this:

my_data = my_data %>% 
  distinct(Sepal.Length, .keep_all = TRUE)

By: Hasan Ayouby

Hasan Ayouby — Thu, 21 May 2020 16:46:24 +0000

How to permanently remove the duplicates? because once i used this function, it acts only like a filter. but the original table stays intact.

By: kassambara

kassambara — Wed, 20 May 2020 13:49:24 +0000

In reply to Moses. Thank you for your feedback, highly appreciated. There is also Outlier identifications

By: Moses

Moses — Wed, 20 May 2020 13:44:33 +0000

You are always on point! A quick one…
What are the major check points in data management? I know there are duplicates, missing data, ….

By: kassambara

kassambara — Wed, 15 Apr 2020 18:42:23 +0000

In reply to Zbig.

If you want to keep distinct rows based on multiple columns, you can go as follow:

library(dplyr)
df <- data.frame(id=c(1,1,1,2,2,2), time=rep(1:3, 2), place=c(1,2,1,1,1,2))
df %>% distinct(id, time, place, .keep_all = TRUE)

By: Zbig

Zbig — Wed, 15 Apr 2020 18:23:02 +0000

Now I have a slightly harder task:
what to do if I want to remove only subsequent, immediate duplicates, but if they are divided by something I want to preserve them.
Example: you have a data frame with object id, time and the place where it happened:
df <- data.frame(id=c(1,1,1,2,2,2), time=rep(1:3, 2), place=c(1,2,1,1,1,2))
and I would like to extract paths of these object – for example object 1 was at place 1, then 2, then back to 1 – and I would like to preserve that in data so that later I can see that it moved from 1 to 2 and then from 2 to 1
any ideas?

By: Andreas Rybicki

Andreas Rybicki — Mon, 20 Jan 2020 07:31:08 +0000

Kassambara,

the lesson “Identify and Remove Duplicate Data in R” was extremely helpful for my task,

Question:
two dataframes like “iris”, say iris for Country A and B,
the dataframes are quite large, up to 1 mio rows and > 10 columns,
I’d like to check, whether a row in B contains the same input in A.
E.g. in ‘iris’ row 102 == 143;
let’s assume row 102 is in iris country_A and row 143 in iris…._B. How could I identify any duplicates in these two DF’s?
I searched in stackexchange but didn’t find any helpful solution.
Thks

By: kassambara

kassambara — Tue, 23 Jul 2019 16:58:44 +0000

In reply to robyn. You can use the following R code:

library(dplyr)
Jan_19 %>% distinct(CON, .keep_all = TRUE)

By: robyn

robyn — Fri, 19 Jul 2019 17:16:45 +0000

hi I’m trying to KEEP ONLY duplicate rows base on a column. I first tested for unique;
unique(Jan_19)
# A tibble: 178,492 x 22

then the number of duplicates base on my CON column
Jan_19[duplicated(Jan_19$CON), ]
# A tibble: 251 x 22

then tried to drop the rows where CON was not duplicated
Jan_19 %>% !distinct(CON, .keep_all = TRUE)
Error in distinct(CON, .keep_all = TRUE) : object ‘CON’ not found

any advise? Thanks for the codes, quite useful

By: kassambara

kassambara — Wed, 03 Jul 2019 19:57:57 +0000

In reply to stonemonroy. Thank you for your positive feedback!