How to Remove Duplicate Lines of Text in JavaScript

To remove duplicate lines in JavaScript, split the text on /\r?\n/, pass the lines through a Set, and join them again: [...new Set(text.split(/\r?\n/))].join('\n').

You need this when cleaning a list of emails or IDs before a bulk import, tidying a log file, merging two CSV exports, or stripping repeats from a word list. The one-liner covers exact duplicates, and the remove duplicate lines tool does it in the browser if you just want the result. For near-duplicates, which differ only in case, whitespace, line endings or blank lines, this article gives a copyable function with a first-wins or last-wins option.

The One-Liner to Remove Duplicate Lines in JavaScript

The shortest way to remove duplicate lines in JavaScript is a Set between split and join:

const dedupeLines = (text) => [...new Set(text.split(/\r?\n/))].join('\n');

dedupeLines('one\nthree\ntwo\nthree\none');
// 'one\nthree\ntwo'

This works because a Set iterates its values in insertion order. Each line stays at the position of its first occurrence, and later copies are dropped. The ES2015 specification defines this order: add appends each new value to the end of the set’s internal list. For clean, machine-generated input, this is all you need. The JavaScript Set methods guide covers the rest of the Set API.

Where Does the Naive Set Version Go Wrong?

A Set compares strings exactly (using SameValueZero), so any difference of even one character makes two lines distinct. Each input below keeps a line you probably meant to drop:

• Case: 'Apple\napple' keeps both lines.
• Trailing whitespace: 'apple \napple' keeps both, because one line ends in a space.
• CRLF line endings: 'apple\r\napple'.split('\n') gives ['apple\r', 'apple']. Splitting on '\n' alone leaves a \r at the end of every line except a final line that has no newline after it, so that last line never matches its earlier copies. The regex separator in split /\r?\n/ fixes this. If you expect text with classic Mac OS line endings (a lone \r), use /\r\n|\r|\n/.
• Blank lines: 'a\n\nb\n\nc' becomes 'a\n\nb\nc'. The second blank line counts as a duplicate of the first, so paragraphs b and c run together.
• Trailing newline: 'a\n\nb\n' becomes 'a\n\nb'. The empty string after the final newline matches the earlier blank line and gets removed.

Avoid lines.filter((l, i, a) => a.indexOf(l) === i). indexOf searches the array linearly, so doing it for every line makes the whole pass O(n²). The spec requires Set lookups to be faster than linear on average, so a Set avoids that rescan. A plain object used as the seen-list has its own problems: a line reading __proto__ or hasOwnProperty can break it.

Dedupe on a Normalised Key, Output the Original Line

To treat near-duplicates as equal, compare a normalised key (the trimmed line, lowercased if you want case-insensitive matching) against the Set, but add the original line to the output.

function removeDuplicateLines(text, options = {}) {
  const {
    caseSensitive = true,
    trim = true,
    collapseWhitespace = false,
    blankLines = 'collapse', // 'keep' | 'collapse' | 'remove'
  } = options;

  const eol = text.includes('\r\n') ? '\r\n' : '\n';
  const trailingNewline = /\r?\n$/.test(text);
  const lines = text.replace(/\r?\n$/, '').split(/\r?\n/);

  const toKey = (line) => {
    let key = trim ? line.trim() : line;
    if (collapseWhitespace) key = key.replace(/\s+/g, ' ');
    return caseSensitive ? key : key.toLowerCase();
  };

  const isBlank = (line) => line.trim() === '';
  const seen = new Set();
  const out = [];

  for (const line of lines) {
    if (isBlank(line)) {
      if (blankLines === 'remove') continue;
      if (blankLines === 'collapse' && out.length && isBlank(out[out.length - 1])) continue;
      out.push(line);
      continue;
    }
    const key = toKey(line);
    if (seen.has(key)) continue;
    seen.add(key);
    out.push(line);
  }

  return out.join(eol) + (trailingNewline ? eol : '');
}

removeDuplicateLines('Apple\napple \n\n\nbanana\nBanana\n', { caseSensitive: false });
// 'Apple\n\nbanana\n'

How the removeDuplicateLines options behave:

• trim uses String.prototype.trim, which also strips no-break spaces and a leading BOM. The output keeps the original indentation.
• caseSensitive: false uses toLowerCase, which gives the same result in every locale.
• blankLines: 'keep' leaves blank lines out of deduplication, so they stay where they are. 'collapse' turns each run of blank lines into one; it checks the last output line, so two blank lines with a dropped duplicate between them still collapse. 'remove' deletes every blank line.
• The function keeps the input’s trailing newline. If any \r\n appears in the input, the output uses \r\n for every line. That is a simplification for files with mixed line endings.

A Map from key to line works the same way: if (!map.has(key)) map.set(key, line). See Map vs Set vs Object for when each structure fits. Unicode normalisation, smart quotes and other paste artefacts belong to a separate step, covered in Cleaning Up Text People Paste Into Your App.

First vs Last Occurrence

When you remove duplicate lines, keeping the first occurrence and keeping the last occurrence give different output. Keeping the last also has two variants that differ in order. For the input ['apple', 'banana', 'Apple', 'cherry'], matched case-insensitively:

function keepLast(lines, toKey = (l) => l.trim().toLowerCase(), { moveToEnd = true } = {}) {
  const byKey = new Map();
  for (const line of lines) {
    const key = toKey(line);
    if (moveToEnd) byKey.delete(key);
    byKey.set(key, line);
  }
  return [...byKey.values()];
}

A Map keeps its keys in the order they were first added, and calling set() on a key it already holds changes the value without moving the key. A pass that only overwrites therefore returns the last version of each line, but at the position where the line first appeared. To place each line where it last appeared, call delete(key) before set(key, line) so the key is added again at the end.

Remove Duplicate Lines From Large Files in the Terminal

For files too big to paste, remove duplicate lines in the shell with sort -u or awk. sort -u removes duplicates by sorting, so the original order is lost. It also compares lines using the current locale’s collation, which in non-C locales can merge lines that are not byte-identical, so use LC_ALL=C sort -u for exact matching.

awk '!seen[$0]++' prints each line the first time it appears and keeps input order, but it stores every unique line in an associative array held in memory. awk '!seen[tolower($0)]++' ignores case. CRLF files carry the \r into both tools, so remove it first with tr -d '\r'.

LC_ALL=C sort -u input.txt > output.txt
awk '!seen[$0]++' input.txt > output.txt
tr -d '\r' < input.txt | awk '!seen[$0]++' > output.txt

Conclusion

Use new Set for exact duplicates. When the input comes from people or several sources, dedupe on a normalised key and output the original line. Choose a blank-line policy and a first-wins or last-wins rule on purpose rather than accepting the defaults. Copy removeDuplicateLines, set the options for your data, and if you want to group related lines instead of removing them, use Object.groupBy.

FAQs

How do I keep only the lines that appear exactly once in JavaScript?

Count occurrences first, then filter. Build a Map of line to count in one pass, then keep lines whose count is 1 with lines.filter((line) => counts.get(line) === 1). This is not deduplication, which keeps one copy of every repeated line. Here, any line that appears two or more times is removed entirely. Count on a normalised key if near-duplicates should count as the same line.

How do I remove lines from one list that already appear in another list?

Load the second list into a Set, then filter the first list against it: const exclude = new Set(listB); listA.filter((line) => !exclude.has(line)). Each check is a Set lookup, so you avoid the quadratic cost of calling includes or indexOf on an array for every line. Split both texts on /\r?\n/ first, and apply the same trim or lowercase key to both sides.

Why do two lines that look identical survive deduplication?

They usually contain different Unicode code points. An accented letter such as é can be stored as one precomposed character or as a plain e followed by a combining accent. A Set compares code units exactly, so it treats the two lines as different. Add line.normalize('NFC') to the key function to match them. trim does not remove zero-width spaces (U+200B), so strip those separately.

How do I remove duplicate lines from a large file in Node.js without reading the whole file into a string?

Stream the file with the node:readline module and check each line against a Set. Create the interface with createInterface({ input: createReadStream(path), crlfDelay: Infinity }), loop over it with for await, and write a line only if the Set does not already contain it. The crlfDelay: Infinity option makes readline treat every CRLF as a single line break. As with awk, memory still grows with the number of unique lines.