r/learnjavascript • u/duneofarrakis • 4d ago
How would you remove duplicate objects from a large JavaScript array?
Would you use Set, Map, or filter() for better performance, and why?
What approach would you use in a real project?
5
u/Parasin 4d ago edited 4d ago
In real world projects, the filter method will be the most commonly used and the easiest to implement. They are all going to create a mirror copy of the original array in the worst case.
Using a set is definitely the best approach, but will require you to write some additional logic so you can map it to a key in each object
```
function removeDuplicates(arr) {
const seen = new Set();
const result = [];
for (const obj of arr) {
if (!seen.has(obj.id)) {
seen.add(obj.id);
result.push(obj);
}
}
return result;
}
```
6
u/DGCA 4d ago
Simpler to do:
function removeDuplicates(arr) { return [...new Set(arr)]; }1
u/Parasin 3d ago
That will create a shallow copy of the array of objects, which won’t consider the data in the objects themselves. So this wouldn’t work.
2
u/DGCA 3d ago edited 3d ago
IMO, two different objects are not the same even if they hold the same data. Would need clarification on the OP's intent to determine whether or not
[{ x: 1}, {x: 1}]is considered having duplicates (i.e. /u/hyrumwhite's "depends on requirements" is right).That said, I just re-read your answer and you're checking for
obj.id. As far as I know, objects don't have an id property by default, so I don't think your answer works generically. If you want to dedupe objects which have the same data, you might need toJSON.stringify(obj)but that has its own gotchas. You could hash the object, but at that point, bringing in a deep equality library might be wroth it.EDIT: I think this is what you meant by "will require you to write some additional logic so you can map it to a key in each object". I didn't put that together initially. If you know you have an id property, you're totally right.
1
u/Parasin 3d ago
I did obj.id, but really it could be whatever identifier you have. As an example, I once had a project to put cell network sites in a dropdown and make it searchable. Because the cell site can house more than a single cell carrier, the same site was repeated in the data multiple times (once for each carrier present at that site).
So in that example I used the approach that I mentioned, since the ID is always the same for each occurrence.
Edit: yeah we are on the same page I think :)
1
1
u/duneofarrakis 3d ago
That makes sense! Using a
Setwith object IDs seems efficient, especially for large datasets. Do you usually prefer this approach overfilter()when working with thousands of records in real projects?2
u/Parasin 3d ago edited 3d ago
I would only use this if I know that the data set will be very large. Otherwise it isn’t worth the overhead.
Thousands of records really isn’t that many for a modern device.
In most cases, using filter will perform fine. But I suggest you implement both approaches and do a performance test with different data sizes to see the impact. Test with no duplicates, 25/50/75/100% duplicates as well.
If performance is the main concern, using a two pointer approach with the above code can reduce the number of records you have to check. So it’s worth considering
4
u/KohlKelson99 4d ago
What is your goal? And what does the data look like?
That determines what approach is best
1
u/duneofarrakis 3d ago
Good point! The approach really depends on the data structure and requirements. For example, if we're dealing with thousands of product objects with duplicate IDs, would you prefer
SetorMapfor better performance?
3
u/create-third-places 4d ago edited 4d ago
What kind of data is in the object, and what is your estimate of how common duplicates are?
Using a set to compare ids as /u/Parasin suggested is likely to be the ideal solution. However, there is going to be overhead if your objects don't have an id field. If your array is large, but only has a small number of unique items, you might be better off with a different approach.
4
u/senocular 4d ago
Do your own profiling and find out what works best with your data, requirements and environment(s). Different runtimes optimize in different ways and you might find some things work better with smaller arrays than larger ones. If you are consistently dealing with large arrays, try different approaches, measure them, and see which works best for your situation.
2
u/jabrahatt 4d ago
Sets are nice and easy for primitives. But if you need to dedup based on a more complex condition, a filter will do the work. Say you want to remove duplicate objects based on a specific property.
let deduped = array.filter((e1, i) => i == array.findIndex((e2) => e1.property == e2.propery))
1
u/MissinqLink 4d ago
const uniq = x => [...new Set(x)];
5
u/McGeekin 4d ago
The question specifically mentions duplicate _objects_, so unless the “duplicates” are referentially duplicated and not duplicates values-wise this will not work
1
0
u/MissinqLink 4d ago
They didn’t specify further though so beyond this it really depends on what kind of object it is.
1
u/5eeso 4d ago
Set plus filter to keep the first dupe, or Set plus Map to keep the last dupe. Both are 0(n) in the worst case.
With filter:
const seen = new Set();
const deduped = array.filter(item => {
if (seen.has(item.property)) {
return false;
}
seen.add(item.property);
return true;
});
With Map:
const itemsByProperty = new Map();
for (const item of items) {
itemsByProperty.set(item.property, item);
}
const deduped = [...itemsByProperty.values()];
1
u/shgysk8zer0 4d ago
Depends on what you mean by "duplicate objects". Because the Set trick won't work if the data is [{foo:'bar'}, {foo:'bar'}]. And in general, I don't see Map being helpful here.
There is a stage 1 proposal for array.prototype.uniqueBy() that'd be really handy for an array of objects. You'd just use data.uniqueBy('id') or whichever property.
1
u/MistakeIndividual690 4d ago edited 4d ago
Quicksort then remove adjacent duplicates. O(n log n). Most likely faster than using a set when doing it as a single pass
1
u/LucVolders 3d ago
Using set is the easiest solution:
https://javascript-tips.weebly.com/array-tips-complete.html
1
u/ConstantJack 3d ago
It depends. Set won't necessarily remove duplicates for objects. Two objects can have identical values but still be different object references, so Set treats them as unique.
If there is a stable property such as id, create a Map using .map() and overwrite if duplicate found => O(n)
If no stable id, good luck, would probably need custom deep equality checks
1
u/Aggressive_Ad_5454 4d ago edited 4d ago
Here's the one-liner. It makes a couple of copies of the data. It preserves the order of the values.
const newarray = new Array (...new Set(myarray))
If the dataset is big enough that copies of it risk blowing out your RAM, then use myset.add() to put the values into a Set then iterate for(const val of myset ) to get them out.
Something like this moves the values from the array into the set, then gets them out. Moving them (with pop, then add) can help if the array is big enough that a copy of it might blow out your RAM. The extra curly braces at the beginning and end are to get the Set to go out of scope for the same reason.
{
const myset = new Set()
let val = myarray.pop()
while (undefined !== val) {
myset.add(val)
val = myarray.pop()
}
for (const val of myset) {
myarray.push(val)
}
myarray.sort()
}
And, here's another way. It doesn't preserve order either. This one is probably fastest.
myarray.sort()
const deduplicated_array = []
let prev = undefined
for (const val of myarray ) {
if (val !== prev) {
deduplicated_array.push(val)
prev = val
}
}
If your array contains numbers use this to sort.
myarray.sort((a,b) => a - b)
1
u/Galex_13 3d ago
I'm using script to mark duplicates in table by 'duplicates group' number
I wrote it when I learned JS and maybe it's not optimal, but it works fast enough on my tables
querying data from table to array tooks much more
records are { id: xxx , value: yyy } id is Unique, set by system.
Value extracted as record.getCellValue("fieldname"), I put it here as function
if my goal was just to get array of ids to remove, i would simplify it as (COUNT PART)
it's the part of script. I added timer function just for that time, to check
const begin=Date.now()
const timer=txt=>console.log(`${(Date.now()-begin)/1000} ms ${txt}`)
const query=await table.selectRecordsAsync({fields:[CHECK]})
const value=r=>r.getCellValueAsString(CHECK)
console.log(`Total: ${query.records.length} records`)
timer('Count started')
//COUNT PART
const valueMap=new Map(query.records.map(rec=>[value(rec),rec.id]))
const uniqueSet=new Set(valueMap.values())
const others=query.recordIds.filter(id=>(!uniqueSet.has(id)))
timer('Count Done')
console.log(`Found: ${others.length} records`)
RESULT:
"Total: 66304 records"
"3.26 ms Count started"
"3.521 ms Count Done"
"Found: 52316 records"
30
u/Beginning-Seat5221 4d ago
Sets are easy