I need to check a JavaScript array to see if there are any duplicate values. How can I do this? I just need to find what the duplicated values are, and I don't actually need their indexes or how many times they are duplicated.
- 31There seems to be years of confusion about what this question asks. I needed to know what elements in the array were duplicated: "I just need to find what the duplicated values are". The correct answer should NOT remove duplicates from the array. That's the inverse of what I wanted: a list of the duplicates, not a list of unique elements.Scott Saunders– Scott Saunders2013-02-22 15:47:29 +00:00Commented Feb 22, 2013 at 15:47
- github.com/lodash/lodash/issues/4852#issuecomment-666366511 I would add this as an answer, but given the length of answers, it would never be seenlonewarrior556– lonewarrior5562020-07-30 14:44:45 +00:00Commented Jul 30, 2020 at 14:44
98 Answers
You could sort the array and then run through it and then see if the next (or previous) index is the same as the current. Assuming your sort algorithm is good, this should be less than O(n2):
const findDuplicates = (arr) => { let sorted_arr = arr.slice().sort(); // You can define the comparing function here. // JS by default uses a crappy string compare. // (we use slice to clone the array so the // original array won't be modified) let results = []; for (let i = 0; i < sorted_arr.length - 1; i++) { if (sorted_arr[i + 1] == sorted_arr[i]) { results.push(sorted_arr[i]); } } return results; } let duplicatedArray = [9, 9, 111, 2, 3, 4, 4, 5, 7]; console.log(`The duplicates in ${duplicatedArray} are ${findDuplicates(duplicatedArray)}`); In case, if you are to return as a function for duplicates. This is for similar type of case.
Reference: What is wrong with my syntax? I am trying to find similar parameters in an array
22 Comments
arr = [9, 9, 9, 111, 2, 3, 3, 3, 4, 4, 5, 7];i++. Instead, they say not to write j = i + +j. Two different things IMHO. I think i += 1 is more confusing than the simple and beautiful i++ :)var sorted_arr = arr.sort() is useless: arr.sort() mutates the original array (which is a problem in its own right). This also discards an element. (Run the code above. What happens to 9?) cc @dystroy A cleaner solution would be results = arr.filter(function(elem, pos) { return arr.indexOf(elem) == pos; })If you want to elimate the duplicates, try this great solution:
function eliminateDuplicates(arr) { var i, len = arr.length, out = [], obj = {}; for (i = 0; i < len; i++) { obj[arr[i]] = 0; } for (i in obj) { out.push(i); } return out; } console.log(eliminateDuplicates([1,6,7,3,6,8,1,3,4,5,1,7,2,6])) Source: Eliminating Duplicates
12 Comments
This is my answer from the duplicate thread:
When writing this entry 2014 - all examples were for-loops or jQuery. JavaScript has the perfect tools for this: sort, map and reduce.
Find duplicate items
var names = ['Mike', 'Matt', 'Nancy', 'Adam', 'Jenny', 'Nancy', 'Carl'] const uniq = names .map((name) => { return { count: 1, name: name }; }) .reduce((result, b) => { result[b.name] = (result[b.name] || 0) + b.count; return result; }, {}); const duplicates = Object.keys(uniq).filter((a) => uniq[a] > 1); console.log(duplicates); // [ 'Nancy' ] More functional syntax:
@Dmytro-Laptin pointed out some code that can be removed. This is a more compact version of the same code. Using some ES6 tricks and higher-order functions:
const names = ['Mike', 'Matt', 'Nancy', 'Adam', 'Jenny', 'Nancy', 'Carl']; const count = names => names.reduce((result, value) => ({ ...result, [value]: (result[value] || 0) + 1 }), {}); // don't forget to initialize the accumulator const duplicates = dict => Object.keys(dict).filter((a) => dict[a] > 1); console.log(count(names)); // { Mike: 1, Matt: 1, Nancy: 2, Adam: 1, Jenny: 1, Carl: 1 } console.log(duplicates(count(names))); // [ 'Nancy' ] 3 Comments
=> syntax.UPDATED: Short one-liner to get the duplicates:
[1, 2, 2, 4, 3, 4].filter((e, i, a) => a.indexOf(e) !== i) // [2, 4] To get the array without duplicates simply invert the condition:
[1, 2, 2, 4, 3, 4].filter((e, i, a) => a.indexOf(e) === i) // [1, 2, 3, 4] Note that this answer’s main goal is to be short. If you need something performant for a big array, one possible solution is to sort your array first (if it is sortable) then do the following to get the same kind of results as above:
myHugeSortedArray.filter((e, i, a) => a[i-1] === e) Here is an example for a 1 000 000 integers array:
const myHugeIntArrayWithDuplicates = [...Array(1_000_000).keys()] // adding two 0 and four 9 duplicates .fill(0, 2, 4).fill(9, 10, 14) console.time("time") console.log( myHugeIntArrayWithDuplicates // a possible sorting method for integers .sort((a, b) => a > b ? 1 : -1) .filter((e, i, a) => a[i-1] === e) ) console.timeEnd("time") On my AMD Ryzen 7 5700G dev machine it outputs:
[ 0, 0, 9, 9, 9, 9 ] time: 22.738ms As pointed out in the comments both the short solution and the performant solution will return an array with several time the same duplicate if it occurs more than once in the original array:
[1, 1, 1, 2, 2, 2, 2].filter((e, i, a) => a.indexOf(e) !== i) // [1, 1, 2, 2, 2] If unique duplicates are wanted then a function like
function duplicates(arr) { return [...new Set(arr.filter((e, i, a) => a.indexOf(e) !== i))] } can be used so that duplicates([1, 1, 1, 2, 2, 2, 2]) returns [1, 2].
When all you need is to check that there are no duplicates as asked in this question you can use the every() method:
[1, 2, 3].every((e, i, a) => a.indexOf(e) === i) // true [1, 2, 1].every((e, i, a) => a.indexOf(e) === i) // false Note that every() doesn't work for IE 8 and below.
4 Comments
[2,2,2,2].filter((e, i, a) => a.indexOf(e) !== i) gives [2, 2, 2]f = arr => [...new Set(arr.filter((e, i, a) => a.indexOf(e) !== i))] can be used so that f([1, 1, 1, 2, 2, 2, 2]) returns [1, 2]Find duplicate values in an array
This should be one of the shortest ways to actually find duplicate values in an array. As specifically asked for by the OP, this does not remove duplicates but finds them.
var input = [1, 2, 3, 1, 3, 1]; var duplicates = input.reduce(function(acc, el, i, arr) { if (arr.indexOf(el) !== i && acc.indexOf(el) < 0) acc.push(el); return acc; }, []); console.log(duplicates); // [1, 3] This doesn't need sorting or any third party framework. It also doesn't need manual loops. It works with every value indexOf() (or to be clearer: the strict comparision operator) supports.
3 Comments
const dupes = items.reduce((acc, v, i, arr) => arr.indexOf(v) !== i && acc.indexOf(v) === -1 ? acc.concat(v) : acc, [])if (arr.indexOf(el) !== i && !acc.includes(el) ) acc.push(el); return acc; works tooconst dupes = items.reduce((acc, v, i, arr) => arr.indexOf(v) === i || acc.includes(v) ? acc : acc.concat(v), [])You can add this function, or tweak it and add it to JavaScript's Array prototype:
Array.prototype.unique = function () { var r = new Array(); o:for(var i = 0, n = this.length; i < n; i++) { for(var x = 0, y = r.length; x < y; x++) { if(r[x]==this[i]) { continue o; } } r[r.length] = this[i]; } return r; } var arr = [1,2,2,3,3,4,5,6,2,3,7,8,5,9]; var unique = arr.unique(); console.log(unique); 3 Comments
The following uses an optimized combined strategy. It optimizes primitive lookups to benefit from hash O(1) lookup time (running unique on an array of primitives is O(n)). Object lookups are optimized by tagging objects with a unique id while iterating through so so identifying duplicate objects is also O(1) per item and O(n) for the whole list. The only exception is items that are frozen, but those are rare and a fallback is provided using an array and indexOf.
var unique = function(){ var hasOwn = {}.hasOwnProperty, toString = {}.toString, uids = {}; function uid(){ var key = Math.random().toString(36).slice(2); return key in uids ? uid() : uids[key] = key; } function unique(array){ var strings = {}, numbers = {}, others = {}, tagged = [], failed = [], count = 0, i = array.length, item, type; var id = uid(); while (i--) { item = array[i]; type = typeof item; if (item == null || type !== 'object' && type !== 'function') { // primitive switch (type) { case 'string': strings[item] = true; break; case 'number': numbers[item] = true; break; default: others[item] = item; break; } } else { // object if (!hasOwn.call(item, id)) { try { item[id] = true; tagged[count++] = item; } catch (e){ if (failed.indexOf(item) === -1) failed[failed.length] = item; } } } } // remove the tags while (count--) delete tagged[count][id]; tagged = tagged.concat(failed); count = tagged.length; // append primitives to results for (i in strings) if (hasOwn.call(strings, i)) tagged[count++] = i; for (i in numbers) if (hasOwn.call(numbers, i)) tagged[count++] = +i; for (i in others) if (hasOwn.call(others, i)) tagged[count++] = others[i]; return tagged; } return unique; }(); If you have ES6 Collections available, then there is a much simpler and significantly faster version. (shim for IE9+ and other browsers here: https://github.com/Benvie/ES6-Harmony-Collections-Shim)
function unique(array){ var seen = new Set; return array.filter(function(item){ if (!seen.has(item)) { seen.add(item); return true; } }); } 6 Comments
var a = ["a","a","b","c","c"]; a.filter(function(value,index,self){ return (self.indexOf(value) !== index )}) 4 Comments
'a' in array, inside filter function the index == 1, whereas self.indexOf('a') == 0This should get you what you want, just the duplicates:
function find_duplicates(arr) { var len=arr.length, out=[], counts={}; for (var i=0;i<len;i++) { var item = arr[i]; counts[item] = counts[item] >= 1 ? counts[item] + 1 : 1; if (counts[item] === 2) { out.push(item); } } return out; } var result = find_duplicates(['one',2,3,4,4,4,5,6,7,7,7,'pig','one']); // -> ['one',4,7] in no particular order. console.log(result); Comments
Find non-unique values from 3 arrays (or more):
ES2015
// 🚩🚩 🚩 🚩 🚩 var arr = [1,2,2,3,3,4,5,6,2,3,7,8,5,22], arr2 = [1,2,511,12,50], arr3 = [22,0], merged, nonUnique; // Combine all the arrays to a single one merged = arr.concat(arr2, arr3) // create a new (dirty) Array with only the non-unique items nonUnique = merged.filter((item,i) => merged.includes(item, i+1)) // Cleanup - remove duplicate & empty items items nonUnique = [...new Set(nonUnique)] console.log(nonUnique) PRE-ES2015:
In the below example I chose to superimpose a unique method on top of the Array prototype, allowing access from everywhere and has more "declarative" syntax. I do not recommend this approach on large projects, since it might very well collide with another method with the same custom name.
Array.prototype.unique = function () { var arr = this.sort(), i=arr.length; // input must be sorted for this to work while(i--) arr[i] === arr[i-1] && arr.splice(i,1) // remove duplicate item return arr } Array.prototype.nonunique = function () { var arr = this.sort(), i=arr.length, res = []; // input must be sorted for this to work while(i--) arr[i] === arr[i-1] && (res.indexOf(arr[i]) == -1) && res.push(arr[i]) return res } // 🚩🚩 🚩 🚩 🚩 var arr = [1,2,2,3,3,4,5,6,2,3,7,8,5,22], arr2 = [1,2,511,12,50], arr3 = [22,0], // merge all arrays & call custom Array Prototype - "unique" unique = arr.concat(arr2, arr3).unique(), nonunique = arr.concat(arr2, arr3).nonunique() console.log(unique) // [1,12,2,22,3,4,5,50,511,6,7,8] console.log(nonunique) // [1,12,2,22,3,4,5,50,511,6,7,8] 7 Comments
var r = []; to get your code working. And worked like charm.r variableThe simplest and quickest way is to use the Set object:
const numbers = [1, 2, 3, 2, 4, 5, 5, 6]; const set = new Set(numbers); const duplicates = numbers.filter(item => { if (set.has(item)) { set.delete(item); return false; } else { return true; } }); // OR more concisely const duplicates = numbers.filter(item => !set.delete(item)); console.log(duplicates); // [ 2, 5 ] 3 Comments
item => !set.delete(item), as Set.delete() return false if the item was not in the set.Using underscore.js
function hasDuplicate(arr){ return (arr.length != _.uniq(arr).length); } 1 Comment
Here's the simplest solution I could think of:
const arr = [-1, 2, 2, 2, 0, 0, 0, 500, -1, 'a', 'a', 'a'] const filtered = arr.filter((el, index) => arr.indexOf(el) !== index) // => filtered = [ 2, 2, 0, 0, -1, 'a', 'a' ] const duplicates = [...new Set(filtered)] console.log(duplicates) // => [ 2, 0, -1, 'a' ] Note:
It works with any numbers including
0, strings and negative numbers e.g.-1- Related question: Get all unique values in a JavaScript array (remove duplicates)The original array
arris preserved (filterreturns the new array instead of modifying the original)The
filteredarray contains all duplicates; it can also contain more than 1 same value (e.g. our filtered array here is[ 2, 2, 0, 0, -1, 'a', 'a' ])If you want to get only values that are duplicated (you don't want to have multiple duplicates with the same value) you can use
[...new Set(filtered)](ES6 has an object Set which can store only unique values)
Comments
Here is my simple and one line solution.
It searches not unique elements first, then makes found array unique with the use of Set.
So we have array of duplicates in the end:
var array = [1, 2, 2, 3, 3, 4, 5, 6, 2, 3, 7, 8, 5, 22, 1, 2, 511, 12, 50, 22]; console.log([...new Set( array.filter((value, index, self) => self.indexOf(value) !== index))] ); Comments
One liner simple way:
var arr = [9,1,2,4,3,4,9] console.log(arr.filter((ele,indx)=>indx!==arr.indexOf(ele))) //get the duplicates console.log(arr.filter((ele,indx)=>indx===arr.indexOf(ele))) //remove the duplicates 3 Comments
indx! do for the first example?indx !== ... - strict inequality.result.filter((ele,indx) => indx !== result.map(e => e.name).indexOf(ele.name));Shortest vanilla JS:
[1,1,2,2,2,3].filter((v,i,a) => a.indexOf(v) !== i) // [1, 2, 2] 1 Comment
new Set(result) will get rid of themvar a = [324,3,32,5,52,2100,1,20,2,3,3,2,2,2,1,1,1].sort(); a.filter(function(v,i,o){return i&&v!==o[i-1]?v:0;}); or when added to the prototyp.chain of Array
//copy and paste: without error handling Array.prototype.unique = function(){return this.sort().filter(function(v,i,o){return i&&v!==o[i-1]?v:0;});} See here: https://gist.github.com/1305056
2 Comments
i&& is for avoiding going out of bounds of the array, but it also means that the first element in the sorted array will not be included. In your example there is no 1 in the resulting array. I.e. return i&&v!==o[i-1]?v:0; should be return v!==o[i-1];Fast and elegant way using ES6 object destructuring and reduce.
It runs in O(n) (1 iteration over the array) and doesn't repeat values that appear more than 2 times:
const arr = ['hi', 'hi', 'hi', 'bye', 'bye', 'asd'] const { dup } = arr.reduce( (acc, curr) => { acc.items[curr] = acc.items[curr] ? acc.items[curr] += 1 : 1 if (acc.items[curr] === 2) acc.dup.push(curr) return acc }, { items: {}, dup: [] }, ) console.log(dup) // ['hi', 'bye'] Comments
Here is a very light and easy way:
var codes = dc_1.split(','); var i = codes.length; while (i--) { if (codes.indexOf(codes[i]) != i) { codes.splice(i,1); } } 1 Comment
Simple code with ES6 syntax (return sorted array of duplicates):
let duplicates = a => {d=[]; a.sort((a,b) => a-b).reduce((a,b)=>{a==b&&!d.includes(a)&&d.push(a); return b}); return d}; How to use:
duplicates([1,2,3,10,10,2,3,3,10]); 1 Comment
ES6 offers the Set data structure which is basically an array that doesn't accept duplicates. With the Set data structure, there's a very easy way to find duplicates in an array (using only one loop).
Here's my code:
function findDuplicate(arr) { var set = new Set(); var duplicates = new Set(); for (let i = 0; i< arr.length; i++) { var size = set.size; set.add(arr[i]); if (set.size === size) { duplicates.add(arr[i]); } } return duplicates; } Comments
With ES6 (or using Babel or TypeScript) you can simply do:
var duplicates = myArray.filter(i => myArray.filter(ii => ii === i).length > 1); 1 Comment
I have just figured out a simple way to achieve this using an Array filter:
var list = [9, 9, 111, 2, 3, 4, 4, 5, 7]; // Filter 1: to find all duplicates elements var duplicates = list.filter(function(value,index,self) { return self.indexOf(value) !== self.lastIndexOf(value) && self.indexOf(value) === index; }); console.log(duplicates); Comments
Higher ranked answers have a few inherent issues including the use of legacy JavaScript, incorrect ordering or with only support for 2 duplicated items.
Here's a modern solution which fixes those problems:
const arrayNonUniq = array => { if (!Array.isArray(array)) { throw new TypeError("An array must be provided!") } return array.filter((value, index) => array.indexOf(value) === index && array.lastIndexOf(value) !== index) } arrayNonUniq([1, 1, 2, 3, 3]) //=> [1, 3] arrayNonUniq(["foo", "foo", "bar", "foo"]) //=> ['foo'] You can also use the npm package array-non-uniq.
Comments
ES5 only (i.e., it needs a filter() polyfill for IE8 and below):
var arrayToFilter = [ 4, 5, 5, 5, 2, 1, 3, 1, 1, 2, 1, 3 ]; arrayToFilter. sort(). filter( function(me,i,arr){ return (i===0) || ( me !== arr[i-1] ); }); 1 Comment
This answer leverages js reduce operator/method to remove duplicates from array:
const result = [1, 2, 2, 3, 3, 3, 3].reduce((x, y) => x.includes(y) ? x : [...x, y], []); console.log(result); 1 Comment
new Set([1, 2, 2, 3, 3, 3, 3]) to remove duplicatesThe following function (a variation of the eliminateDuplicates function already mentioned) seems to do the trick, returning test2,1,7,5 for the input ["test", "test2", "test2", 1, 1, 1, 2, 3, 4, 5, 6, 7, 7, 10, 22, 43, 1, 5, 8]
Note that the problem is stranger in JavaScript than in most other languages, because a JavaScript array can hold just about anything. Note that solutions that use sorting might need to provide an appropriate sorting function—I haven't tried that route yet.
This particular implementation works for (at least) strings and numbers.
function findDuplicates(arr) { var i, len=arr.length, out=[], obj={}; for (i=0;i<len;i++) { if (obj[arr[i]] != null) { if (!obj[arr[i]]) { out.push(arr[i]); obj[arr[i]] = 1; } } else { obj[arr[i]] = 0; } } return out; } Comments
Using includes to test if the element already exists:
var arr = [1, 1, 4, 5, 5], darr = [], duplicates = []; for(var i = 0; i < arr.length; i++){ if(darr.includes(arr[i]) && !duplicates.includes(arr[i])) duplicates.push(arr[i]) else darr.push(arr[i]); } console.log(duplicates); <h3>Array with duplicates</h3> <p>[1, 1, 4, 5, 5]</p> <h3>Array with distinct elements</h3> <p>[1, 4, 5]</p> <h3>duplicate values are</h3> <p>[1, 5]</p>